3 days from now (actually 2 days 22 hours and 30 minutes) I'll be a married man. This can at least partially explain the lack of writing.
Last week I tried to find a method to write an Escel file from C#. This thing appears to be a bigger problem than can be expected mainly because the direct method, using OLE, is SLOW. I used it for reading the file, 10 sheets of 4 columns and 600 lines. It takes approximately 10 minutes using the OLE. That's a NO-GO! So we started digging and found that Excel readers are not so rare. Though not so common as one would think they ought to be. We took one and voila, it takes 5 seconds to load and display. Yep, displaying such a large dataset takes a bit of time.
As for the writing, we're still searching. We've found a tool to convert the XMLs of the office2007 to the Excel 2000 we use. But writing the XMLs is problematic, it appears that M$ keeps things as nasty as possible even in places they can't be.
So at this moment we're still looking for solutions.
BTW there are commercial components that do the job, but we want the control over the resulting format, after a minor error with the reader we have all the reasons to dig into the source-code in the writing component.
That's for now, and I'll write again after the marriage, probably at the beginning of 2008:)
Monday, December 17, 2007
Tuesday, December 4, 2007
"Date/Time" field in Access
There are times when you need a small DB which is a single file. Usually I prefer the FireBird, but, when the boss says "Niet" you have to obey. So I had to create the DB using Access, actually it consists of 3 tables and 2 foreign keys. Easy enough for Access to handle.
Next thing I had to connect to it, that's also an easy job:
const DB_LOCATION = "c:\Data\TimesDB.mdb"
Dim conn : Set conn=CreateObject("ADODB.Connection")
conn.Open "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & DB_LOCATION & ";Persist Security Info=False"
So far there's nothing new or any problems. But the troubles begin once "Date/Time" field is involved.
We write the dates as "dd/MM/yy" or "dd/MM/yyyy". The server on the other hand is configured to use dates made by an unknown nature (if it was "MM/dd/yyyy" it would be fine). The result is that every time I've inserted a field the value was "computer-dependant". Obviously it's not the way things are meant to be.
And for this reason I made myself a small useful function:
function parseDate(tmpDate)
if instr(1, tmpDate, "/")>0 then
dim tmpDay
dim tmpMonth
dim tmpYear
dim i
i = instr(1, tmpDate, "/")
tmpDay = mid(tmpDate, 1, i - 1)
tmpDate = mid(tmpDate, i + 1)
i = instr(1, tmpDate, "/")
tmpMonth = mid(tmpDate, 1, i - 1)
tmpYear = mid(tmpDate, i + 1)
if len(tmpYear)=2 then tmpYear = "20" & tmpYear
parseDate = tmpYear & "-" & tmpMonth & "-" & tmpDay
else
parseDate = tmpDate
end if
end function
Note the returned value is in the ISO format (yyyy-MM-dd) this format is a well known format, thus making it the best choice.
P.S. This function is useful in FireBird too, though it's not a must when you can actually "force" the DB to use formats in the creation script. A thing that is not recomended when using Access.
Next thing I had to connect to it, that's also an easy job:
const DB_LOCATION = "c:\Data\TimesDB.mdb"
Dim conn : Set conn=CreateObject("ADODB.Connection")
conn.Open "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & DB_LOCATION & ";Persist Security Info=False"
So far there's nothing new or any problems. But the troubles begin once "Date/Time" field is involved.
We write the dates as "dd/MM/yy" or "dd/MM/yyyy". The server on the other hand is configured to use dates made by an unknown nature (if it was "MM/dd/yyyy" it would be fine). The result is that every time I've inserted a field the value was "computer-dependant". Obviously it's not the way things are meant to be.
And for this reason I made myself a small useful function:
function parseDate(tmpDate)
if instr(1, tmpDate, "/")>0 then
dim tmpDay
dim tmpMonth
dim tmpYear
dim i
i = instr(1, tmpDate, "/")
tmpDay = mid(tmpDate, 1, i - 1)
tmpDate = mid(tmpDate, i + 1)
i = instr(1, tmpDate, "/")
tmpMonth = mid(tmpDate, 1, i - 1)
tmpYear = mid(tmpDate, i + 1)
if len(tmpYear)=2 then tmpYear = "20" & tmpYear
parseDate = tmpYear & "-" & tmpMonth & "-" & tmpDay
else
parseDate = tmpDate
end if
end function
Note the returned value is in the ISO format (yyyy-MM-dd) this format is a well known format, thus making it the best choice.
P.S. This function is useful in FireBird too, though it's not a must when you can actually "force" the DB to use formats in the creation script. A thing that is not recomended when using Access.
Monday, November 19, 2007
I decided to write this entry after seeing an article in codeproject describing the "disadvantage" of recursion. I won't put a link to the article because it's not an article worthy of reading, too much programming "don't dos".
Lets take for example the common use (i.e. the one every computer class use) calculating n!
the for loop looks like:
res = 1;
for(i=1;i<=n;i++) {
res *= i;
}
and the function looks like:
function calc(int n)
{
if (n>1) {
return calc(n-1)*n;
} else {
return 1;
}
}
The function will have to assign a new space for n in each iteration. That's a time consuming operation, but if I'll write the function:
function calc(ref int n)
{
if (n>1) {
return n*calc(--n);
} else {
return 1;
}
}
it would work faster, though no one would have the ability to decipher this call...
It looks like a loop is better than recursion because of the stack assignments, but that's incorrect. Basically it's a question of what's more appropriate for the problem at hand.
A loop is easier to handle for as long as it's a "short" calculation, by short I mean the number of lines. A recursive function is better for the cases where the operation is complicated, or if there are memory allocations you have to clear within the operation (in the .NET one can use GC.Collect(), but other languages don't have such a mechanism.)
Summarizing this article in one line:
You're a programmer, THINK BEFORE WRITING CODE!
Lets take for example the common use (i.e. the one every computer class use) calculating n!
the for loop looks like:
res = 1;
for(i=1;i<=n;i++) {
res *= i;
}
and the function looks like:
function calc(int n)
{
if (n>1) {
return calc(n-1)*n;
} else {
return 1;
}
}
The function will have to assign a new space for n in each iteration. That's a time consuming operation, but if I'll write the function:
function calc(ref int n)
{
if (n>1) {
return n*calc(--n);
} else {
return 1;
}
}
it would work faster, though no one would have the ability to decipher this call...
It looks like a loop is better than recursion because of the stack assignments, but that's incorrect. Basically it's a question of what's more appropriate for the problem at hand.
A loop is easier to handle for as long as it's a "short" calculation, by short I mean the number of lines. A recursive function is better for the cases where the operation is complicated, or if there are memory allocations you have to clear within the operation (in the .NET one can use GC.Collect(), but other languages don't have such a mechanism.)
Summarizing this article in one line:
You're a programmer, THINK BEFORE WRITING CODE!
Sunday, November 11, 2007
IP Mask Merging
I'm back after a week vacation. I was drafted to do nothing for a week, apparently I do to much work.
The issue for today is: IP mask, or how to merge a few subnets into a big subnet.
Easy said, not as easily done. The function I had to make has simple idea, "we have a range of IPs (startIP to endIP) and we want a subnet for it."
The resulting algorithm, I had to devise it after a failed algorithm didn't do the job, is as follows:
1. check the startingIP min bits (i.e. 192.168.0.0 is 192.168.0.0/13)
2. if the endIP matches the region made by this bits - hooray, we're done! if not move to 3.
3. if the endIP is bellow the available range - increase the bits, move to 2 else move to 4.
4. add the current subnet, and start the whole process with the range (startIP + (2 power (32-bits)) to endIP
Sorry I can't post the source code for copyright reasons, we're using it after all...
Important notes:
If you're using an IP to number conversion make sure it's a 4 byte unsigned (uint32) or in the case of vb/vbs use double (or force the result to be double by using cdbl(...))
Another point of interest is the "check the startingIP min bits" it's a bit wise comparison of the IP, a co-worker of mine used the following method:
find the relevant byte (byte1.byte2.byte3.byte4/bits) "AND" it with 256-(2 power (bits mod 8)), it's needed especially if you can't use a uint32.
The issue for today is: IP mask, or how to merge a few subnets into a big subnet.
Easy said, not as easily done. The function I had to make has simple idea, "we have a range of IPs (startIP to endIP) and we want a subnet for it."
The resulting algorithm, I had to devise it after a failed algorithm didn't do the job, is as follows:
1. check the startingIP min bits (i.e. 192.168.0.0 is 192.168.0.0/13)
2. if the endIP matches the region made by this bits - hooray, we're done! if not move to 3.
3. if the endIP is bellow the available range - increase the bits, move to 2 else move to 4.
4. add the current subnet, and start the whole process with the range (startIP + (2 power (32-bits)) to endIP
Sorry I can't post the source code for copyright reasons, we're using it after all...
Important notes:
If you're using an IP to number conversion make sure it's a 4 byte unsigned (uint32) or in the case of vb/vbs use double (or force the result to be double by using cdbl(...))
Another point of interest is the "check the startingIP min bits" it's a bit wise comparison of the IP, a co-worker of mine used the following method:
find the relevant byte (byte1.byte2.byte3.byte4/bits) "AND" it with 256-(2 power (bits mod 8)), it's needed especially if you can't use a uint32.
Thursday, October 25, 2007
SharperCV (the .NET wrapper of OpenCV)
This week I had to deal with OpenCV. An old open-source library from Intel which deals with image processing. The "old" means it must be wrapped for usage inside C#, I've found SharperCV does the job and skipped the writing of C++ wrapper.
Next step is compare 2 images.
Welp, it appears that comparing 2 images is not as trivial as you might think, first thing to do is make sure they have the same scale (obvious, but not ensured.) This barrier is non-existent when comparing the image to itself.
After seeing results from the "trivial" comparison I moved on.
The second thing to do is define the comparison method, in the case I was responsible for we chose the easy ones: Xor, optical flow.
Xor is a very good indication as to whether the images are the same or not, low (enough) Xor.sum means the images are identical (too the lowness degree.) Yet even the static frame of "error" has a value larger than 0 for its Xor.
The code for the Xorring:
SharperCV.CvImage CurImage = new SharperCV.CvImage("input1.jpg");
SharperCV.CvImage PrevImage = new SharperCV.CvImage("input2.jpg");
SharperCV.CvImage Xorred = CurImage.Xor(PrevImage, null);
double XorredAlpha = Xorred.Sum().val0;
Optical flow, on the other hand returns a good indication to whether there's a change, but a black frame turning to white frame results in "no change".
A most important thing to do is turn the images to grey, otherwise a nice error message appears.
And the code is:
SharperCV.CvImage greyCurImg = new SharperCV.CvImage(CurImage.Size, SharperCV.BitDepths.IPL_DEPTH_8U, 1);
SharperCV.CvImage greyPrevImg = new SharperCV.CvImage(PrevImage.Size, SharperCV.BitDepths.IPL_DEPTH_8U, 1);
SharperCV.CvImage BMvelx = null;
SharperCV.CvImage BMvely = null;
greyCurImg.CalcOpticalFlowBM(greyPrevImg, blockSize, shiftSize, maxRange, out BMvelx, out BMvely);
I've used the following definitions, but that's cause "it works".
SharperCV.CvSize blockSize = new SharperCV.CvSize(16, 16);
SharperCV.CvSize shiftSize = new SharperCV.CvSize(4, 4);
SharperCV.CvSize maxRange = new SharperCV.CvSize(16, 16);
Be careful, the resulting images are small (we did some "blocking"), but the real value lies in the average, which gives the indication of "change".
Of course fine tuning is required for the results, but that's a must in almost every statistical system.
Next step is compare 2 images.
Welp, it appears that comparing 2 images is not as trivial as you might think, first thing to do is make sure they have the same scale (obvious, but not ensured.) This barrier is non-existent when comparing the image to itself.
After seeing results from the "trivial" comparison I moved on.
The second thing to do is define the comparison method, in the case I was responsible for we chose the easy ones: Xor, optical flow.
Xor is a very good indication as to whether the images are the same or not, low (enough) Xor.sum means the images are identical (too the lowness degree.) Yet even the static frame of "error" has a value larger than 0 for its Xor.
The code for the Xorring:
SharperCV.CvImage CurImage = new SharperCV.CvImage("input1.jpg");
SharperCV.CvImage PrevImage = new SharperCV.CvImage("input2.jpg");
SharperCV.CvImage Xorred = CurImage.Xor(PrevImage, null);
double XorredAlpha = Xorred.Sum().val0;
Optical flow, on the other hand returns a good indication to whether there's a change, but a black frame turning to white frame results in "no change".
A most important thing to do is turn the images to grey, otherwise a nice error message appears.
And the code is:
SharperCV.CvImage greyCurImg = new SharperCV.CvImage(CurImage.Size, SharperCV.BitDepths.IPL_DEPTH_8U, 1);
SharperCV.CvImage greyPrevImg = new SharperCV.CvImage(PrevImage.Size, SharperCV.BitDepths.IPL_DEPTH_8U, 1);
SharperCV.CvImage BMvelx = null;
SharperCV.CvImage BMvely = null;
greyCurImg.CalcOpticalFlowBM(greyPrevImg, blockSize, shiftSize, maxRange, out BMvelx, out BMvely);
I've used the following definitions, but that's cause "it works".
SharperCV.CvSize blockSize = new SharperCV.CvSize(16, 16);
SharperCV.CvSize shiftSize = new SharperCV.CvSize(4, 4);
SharperCV.CvSize maxRange = new SharperCV.CvSize(16, 16);
Be careful, the resulting images are small (we did some "blocking"), but the real value lies in the average, which gives the indication of "change".
Of course fine tuning is required for the results, but that's a must in almost every statistical system.
Thursday, October 18, 2007
Flash - from a programmer's view
As of yesterday I'm a "flash programmer", since it's the first time I've touched flash I had some difficulties. The first is the dev-env, Adobe does (almost) everything to be "not like M$". In some cases it hurts... P.S. if somebody knows what's the keys for the stepping in the debug mode (F10 & F11 in the VS) let me know...
The second thing was an interesting "feature": XML is loaded in the background (async = true), but there's no telling when it finished loading.
The bypass is use a "timer", I used the following code:
//Load the XML
_global.setTimeout(TimeoutFunc,1000,"");
function TimeoutFunc() {
if (my_xml.loaded) {
//The XML was loaded...
} else {
//Check again, you might wish to add a counter for the cases
//Execution must stop and not re-evaluate it every second.
_global.setTimeout(TimeoutFunc,1000,"");
}
}
The strange thing is that a 100ms timeout makes the debugger crash! And with the "feature" of not saving the other files before running them. I swore like sailor on a bad day.
Besides this effect, flash looks like a reasonable scripting enviroment. It's not a real programming heaven, but it's certainly not HELL like VB6. Though a better support for some features would be good.
The second thing was an interesting "feature": XML is loaded in the background (async = true), but there's no telling when it finished loading.
The bypass is use a "timer", I used the following code:
//Load the XML
_global.setTimeout(TimeoutFunc,1000,"");
function TimeoutFunc() {
if (my_xml.loaded) {
//The XML was loaded...
} else {
//Check again, you might wish to add a counter for the cases
//Execution must stop and not re-evaluate it every second.
_global.setTimeout(TimeoutFunc,1000,"");
}
}
The strange thing is that a 100ms timeout makes the debugger crash! And with the "feature" of not saving the other files before running them. I swore like sailor on a bad day.
Besides this effect, flash looks like a reasonable scripting enviroment. It's not a real programming heaven, but it's certainly not HELL like VB6. Though a better support for some features would be good.
Thursday, October 11, 2007
Those are the SMILS, I forgot the brackets hiding by the browser...
SMIL1
smil
seq repeatCount="1000"
excl
priorityClass id="Priority1" peers="Stop" lower="Defer"
seq begin="wallclock(2007-10-11T00:00:00)"
media src="PATH_TO_FILE"/
...
/seq>
/priorityClass
/excl
/seq
/smil
SMIL2
smil
seq repeatCount="1000"
excl
priorityClass id="Priority1" peers="Stop" lower="Defer"
seq begin="wallclock(2007-10-11T00:00:00)"
media src="PATH_TO_FILE"/
...
/seq>
/priorityClass
priorityClass id="Priority2" peers="Stop" lower="Defer"
seq
media src="PATH_TO_FILLER"/
...
/seq>
/priorityClass
/excl
/seq
/smil
SMIL1
smil
seq repeatCount="1000"
excl
priorityClass id="Priority1" peers="Stop" lower="Defer"
seq begin="wallclock(2007-10-11T00:00:00)"
media src="PATH_TO_FILE"/
...
/seq>
/priorityClass
/excl
/seq
/smil
SMIL2
smil
seq repeatCount="1000"
excl
priorityClass id="Priority1" peers="Stop" lower="Defer"
seq begin="wallclock(2007-10-11T00:00:00)"
media src="PATH_TO_FILE"/
...
/seq>
/priorityClass
priorityClass id="Priority2" peers="Stop" lower="Defer"
seq
media src="PATH_TO_FILLER"/
...
/seq>
/priorityClass
/excl
/seq
/smil
This is my BLOG's initial post:)
I'll deal today with one of the most annoying tasks: converting a bunch of excel shits (sheets is not the correct term for excel) to a SMIL.
First thing to do is "standardize the format", excel is a large grid (2D array) while the SMIL is an hierarchical XML. This means that a certain column(s) in the excel transforms to a certain value in the SMIL.
In this case I used the sheets name as the date, column A as the hour, and column B as the file to play. The resulting SMIL should be of this format:
...
The excel code (for creating the sequences) is:
i=2
while range("A1").cells(i,2).text<>""
if range("A1").cells(i,1).text<>"" then
res = res & ""
res = res & ""
end if
res = res & "
wend
res = mid(res, 7) 'Remove the first " "
NOTE:
The SMIL format presented earlier might appear to hang at times. This is due to the fact no "fillers" were supplied. Usually where I work we solve it by using a SMIL of this type:
...
...
In this case when the priority1 items stop to play the priority2 item continues it's play. The priority1 items enter when the relevant event is fired (in the SMILs in this post it's an exact time)
I'll deal today with one of the most annoying tasks: converting a bunch of excel shits (sheets is not the correct term for excel) to a SMIL.
First thing to do is "standardize the format", excel is a large grid (2D array) while the SMIL is an hierarchical XML. This means that a certain column(s) in the excel transforms to a certain value in the SMIL.
In this case I used the sheets name as the date, column A as the hour, and column B as the file to play. The resulting SMIL should be of this format:
...
The excel code (for creating the sequences) is:
i=2
while range("A1").cells(i,2).text<>""
if range("A1").cells(i,1).text<>"" then
res = res & ""
res = res & "
end if
res = res & "
wend
res = mid(res, 7) 'Remove the first "
NOTE:
The SMIL format presented earlier might appear to hang at times. This is due to the fact no "fillers" were supplied. Usually where I work we solve it by using a SMIL of this type:
...
...
In this case when the priority1 items stop to play the priority2 item continues it's play. The priority1 items enter when the relevant event is fired (in the SMILs in this post it's an exact time)
Subscribe to:
Posts (Atom)