The first ADC conversion result after switching reference voltage source may be inaccurate, and the user is advised to discard this result.

Knowing that the Arduino libraries don’t actually set the reference voltage until the time the first sample is made, it seemed logical that these first samples were an artifact of the voltage changing. However, it wasn’t just the first sample, or the first couple samples. It could take dozens of samples before the value returned had stabilized (i.e. stopped increasing by more than 2 LSB). Being a boot-only issue. I was content to ignore it until I started trying to read the on-chip temperature sensor.

You don’t hear a lot about it but ATmegas have an on-chip temperature sensor connected to their ADC MUX. It can only be read if the analog reference is set to use the 1.1V internal voltage reference. All my other analog inputs needed to be referenced to the 3.3V AVCC, so my code needed to switch back and forth. Easy enough to switch to the internal reference and link the ADCMUX to the temperature sensor, which is ADC8

// Switch to ADC8 and analog reference INTERNAL
ADMUX = (INTERNAL << 6) | 8;

The problem of the first N reads now being inaccurate was now being realized on every cycle. This phenomenon finally warranted investigation.

When the ATmega is running on DEFAULT (AVCC) or INTERNAL voltage reference, the AREF pin is connected to these sources so it may therefore be decoupled by an external capacitor to improve noise immunity. This is the sort of thing you see in many schematics.

A capacitor attached to AREF for the purpose of reducing noise

Adding capacitance to this pin means from switching from the 3.3V of AVCC to the 1.1V of the internal reference is no longer a nearly instantaneous switch. Here's a scope graph of switching from DEFAULT at 3.3V to EXTERNAL (which has no source attached) to INTERNAL (1.1V) and back to DEFAULT. Given 2.5ms divisions it takes over 10ms to decay down to near-0V and at least 2.5ms to go from DEFAULT to INTERNAL.

Analog Reference switching time with a 0.1uF capacitor at 3.3V

Given that we know the capacitor is 0.1uF and the starting voltage is 3.3V, if we count the time it takes to lose 63% of its charge (down to 1.221V) we can calculate the internal resistance of the the AVR and therefore predict the amount of time it takes to switch from 3.3V to 1.1V directly. In this case it takes 3.353ms

3.353ms = R * 0.1uF
3.353x10-3s / 1.0x10-7 = R
R = 33530 ohms

From the equation V = V0 * e ^ (-t/RC)
t = -1n(V/V0) * RC
t = -ln(1.1/3.3) * 33530 * 1.0x10-7
t = 3.68x10-3 = 3.68ms

This is fairly unacceptable for my application considering I wanted the whole operation to be completed in a lot less time than that. Lowering the AREF capacitor would proportionally lower the discharge time

0.1uF = 3.68ms
10nF = 368us
1nF = 36.8us

It is important to note that this is also the same effect I was seeing on boot up. When you first start the ATmega the reference is set to EXTERNAL so there is 0V on the AREF pin. Arduino libraries will switch the reference to DEFAULT on your first analogRead() call unless you explicitly change it using analogReference(). The AREF capacitor now must charge to AVCC before you can take an accurate reading. Fortunately the resistance between the two is very small so it charges on the order of nanoseconds. Still, there can be a small difference in your ADC reads until the capacitor charges.

Back to the task at hand, measuring the on-chip temperature sensor:

unsigned int getAvrTemp(void)
{
    unsigned char high, low;
    ADMUX = (INTERNAL << 6) | 8;
    // delay goes here if need
    ADCSRA |= bit(ADSC);
    while (ADCSRA & bit(ADSC));
    low = ADCL;
    high = ADCH;
    return (high << 8) | low;
}

“Due to process variation” the value you get back is initially worthless. Both the internal reference voltage and the temperature circuit have some amount of tolerance and temperature variation coefficients. For example the “Typical Case” unit should read 314mV at 25C and 380mV at 85C. My device read 373mV indicating my living room was about 78C. To get accurate values you must store some calibration data specific to your ATmega unit. For more information on calibration see Calibration of the AVR's internal temperature reference

I ordered the Arduino Duemilanove and buttons from SparkFun, the WiShield 2.0 from AsyncLabs, and the components and blower from DigiKey. What you end up with is a whole dining room table full of parts.

It helps to not be married if your dining room table is going to look like this for a week.

Testing the LCD controller with some simple temperature measurements

I spent a lot of time on the breadboarding trying to work out some hardware issues. The temperature probes are designed such that there’s one wire inside them and the metal on the outside is the other conductor. This was giving me all sorts of erratic measurements if two of probe wires came in contact with each other. I tried numerous things to prevent this but was never 100% successful in isolating the two circuits. I finally gave up and went with using the outside as ground because it gave better results and figured I’d come back and revisit the problem after I had a working device. Bob’s design used the shield as ground, and the Maverick digital thermometer I stole the probes from also used shield as ground, so I assumed that was the best design. Looking back at this from the future I can say there does not appear to be this issue at all in the final device. I also took some time to get organized.

Lauren Murray Signature Series Tackle Box

Once I was relatively certain the key subsystems were working properly I needed to move to board layout. The free version of Eagle allows you to do board layouts too, as long as you don’t try to drop a component outside the free version design space square. I needed compactness to fit inside the project box I had bought, so things needed to be tight without getting too complex because I am an amateur solderer working on cheap Radio Shack perfboard. Here’s my design, but note there are no inputs here. I ran the inputs along the top of the board in the final design.

HeaterMeter Board Layout v1

Then came the soldering. I used strips of wire from an old 40pin IDE cable, which are small stranded wires, but even with these small wires the underside became a mess. Since then I’ve heard the suggestion to use teflon insulator or “POLYTHERMALEZEtm” 30AWG wire, which is smaller solid core wire that should be easier to work with. When I build the next version of HeaterMeter, I will use that wire and probably wirewrap most of the connections. Building this was like a nightmare with cutting just the right length wires then soldering without making a complete mess or wiring something backward because I flipped it upside down.

Wiring HeaterMeter over the course of 5 hours

Luckily, I only swapped two connections throughout the whole construction so I didn’t have to spend much time debugging. I had the LCD contrast pot hooked up such than when you turned the contrast all the way down it would be a short to ground and the whole device would power off. Thank goodness the Arduino has a fuse in it to prevent me from blowing my USB port. Here we are reading the 4 temperature probes.

Wiring complete showing 4 temperature probes and their ADC values

Physical construction was my next hurtle– trying to keep things as professional as possible and not just stuffing everything into a shoebox.

Drill and file out holes for the USB and 12v power input

Drill, jigsaw, and a lot of filing to make the LCD and button holes

Construction complete!

Everything fit inside like a glove. I had intended the LCD to go across the short dimension but it didn’t fit and the project box was like $10 so I wasn’t going to throw it away. It isn’t objectionable in this configuration though.

The guts of HeaterMeter v3

The pit blower fan would also need a mechanism for holding it to the grill. I designed a simple box in Google Sketchup that I prototyped in cardboard. I originally imagined it as cut from sheet metal but that’s hard so I’m still using the cardboard mockup at this time. It is showing some wear but hasn’t caught fire or anything. I connected the fan to HeaterMeter using an old RCA composite video cable.

Fan box mechanical drawing

Here’s what the whole thing looks like in operation. The photos are from an early “Burn Test” from before the case was completed, buttons were built, and WiFi card ordered.

HeaterMeter controlling the Big Green Egg

PID Grill Controller project success

You may have noticed there is software on the device. I haven’t covered that yet but it deserves its own post.

The Arduino Duemilanove Hadware Platform

• ATmel ATmega168 Micocontroller
• RISC 8-bit CPU running at 16MHz / 5V
• 32kB code space (30,720 bytes usable)
• 2kB RAM
• 1kB Flash storage EEPROM
• Integrated ADC, Digital PWM, SPI bus interface, UART TTL Serial
• C/C++ programming environment

Yeah 2kB of RAM. For comparison, my blog front page is currently 250 times that. 30kB of usable code space? The default Delphi Project1.exe baseline with no user code is 13 times as large. Squeezing everything in to that space seems like a challenge: a web server, LCD and buttons with a menuing system, and I reckon it should control a grill too.

This post was going to be about the platform, but that’s boring. Let’s just jump into the project and I’ll introduce things as needed.

Designing HeaterMeter

Usually the first step of any software design project is coming up with the project name. You can prove this theory by going into Visual Studio and selecting File | New | Project. BAM. It is prompting you for the name. Nothing says “Version 1.0″ like just accepting the default name of “Project1″. Consider using something edgy that shows a lot of time went into development, like “Project10″ or something. Using numbers to name everything seems convenient, but can really get confusing if you use this method to name everything you create, or pets you own. As luck would have it, this article is being written after the device has already been built, so we have the luxury of already having the project named by a friend of mine. The project will be called HeaterMeter, or HeMe, or just “that grill control thing I’ve spent 100 hours on”.

I wasn’t sure I was up for completing this project, so I belayed the ordering of the WiFi card (WiShield 2.0), by far the most expensive component, until I had a working prototype. Knowing I’d probably be using it there were certain design changes from Bob’s design.

Bob used the original WiShield 1.0. The 2.0 adds a 16Mbit dataflash chip to the design, which is accessible by the SPI bus. The idea is that dataflash allows you to serve some static content web pages or store log data in a non-volatile area. That’s actually a good thing, considering the diminutive size of the program space available on the Arduino. The WiFi card itself already uses a large number of digital pins, and this 2.0 version adds another:

• 1 WiFi LED. Indicates the WiFi is connected, but can be disabled
• 1 Interrupt line. One digital line is used as an interrupt. The Arduino can watch this line and call an Interrupt Service Routine (ISR) in your code when it changes, just like a regular operating system does.
• 1 SPI Clock
• 2 SPI Data lines, called Master In Slave Out (MISO) and Master Out Slave In (MOSI)
• 2 SPI Slave Select, this is an active-low line that indicates what device on the SPI bus you’re talking to. The WiFi needs one and the dataflash on the shield needs one too.

In addition to those used by the wireless, the Arduino itself and our HeaterMeter parts need lines too.

• 2 for Arduino serial Rx/Tx. The Arduino talks and is programmed over serial (or in the case of the Duemilanove, a FTDI USB to Serial) so these should remain unused by our application.
• 4 LCD Data (D0-D3). The LCD uses a Hitachi HD44780 parallel interface, which can receive a whole 8 bits at a time but Bob’s design uses the 4 bit mode, which shifts data in a nibble at a time
• 1 LCD Register Select (RS). When this is low, the LCD interprets data as commands. When it is high, it interprets it as character data
• 1 LCD Enable (E). I prefer to call this EXECUTE because when you set this pin, the LCD acts on the data currently on the other pins. This allows you to set the data and RS pins individually with each intermediate state not making your LCD all wonky in between.
• 1 pin for the HeaterMeter PWM fan output.
• At least 2 digital pins for button inputs to adjust the temerature

Sweet, so 18 digital I/O pins and the Arduino has 14 so we’re all se… crap.

Slimming the LCD

Aside from the entire WiFi shield, the next largest consumer of digial I/O lines is the Hitachi HD44780 parallel LCD controller. Because the LCD doesn’t care what the value of the data and RS lines are until the E line is toggled, we can use this to our advantage by shifting the data in one bit at a time. To perform this, I chose an 74LS164 parallel shift register based on this ingenious 2-wire method. The clever bit is that he uses a resistor and a diode to form an AND gate that allows you to use the data line as both data and E. Even better, someone else had also written the C++ class for the Arduino.

Myke Predko's 2-wire LCD Controller

ZOMG Buttonz

The first thing we can do use the analog ADCs for reading the button input. On the Arduino, the 6 analog inputs can actually be used for digital I/O as well, but I’ll need at least two of those for reading temperature probes. Already I was considering expandability. What if I wanted add more temperature probes, or more buttons? I’d hate to paint myself into that corner. I decided to use a bunch of buttons with resistors to form a voltage divider and read N buttons from one analog input. I spent a good amount of time in the java circuit simulator screwing around with different resistor values, trying to find a good combination of values that allowed me to detect if two buttons are pressed at the same time. That lasted for about an hour until I gave up and decided that with four buttons I wouldn’t need to define button chords. Another principle of development is that if at first you don’t succeed change the rules.

A 3-wire circuit (5V,Gnd,Data) for reading buttons using an analog input

Final Digital I/O Layout

With the LCD converted from six to two wire and the buttons moved to an analog circuit, we actually have two pins spare. I’ve arranged them such that they are Pulse Width Modulation (PWM) control pins to that they may be used to control a second and third blower if the system is to be expanded to support multiple grills in the future. Here’s the final digital pin layout.

D00 - Rx
D01 - Tx
D02 - WiFi Int
D03 (PWM) - Blower
D04 - LCD Clk
D05 (PWM) -
D06 (PWM) -
D07 - SS for flash
D08 - LCD Data/Enable
D09 (PWM) - WiFi LED
D10 (PWM) - SS for WiFi
D11 (PWM) - SPI MOSI
D12 - SPI MISO
D13 - SPI CLK

Silence the Blower

The basic operating principle of HeaterMeter to vary the speed of a fan to provide air for combustion in the grill. When the fan is off there should be little to no air getting into the grill and the fire starts to go out. To achieve this, we’ll have to close up the grill as much as possible and that means that when the fan is on, there will be a high static pressure inside the grill as it tries to force air into a mostly-sealed chamber. Based on this operating principle, a blower motor was chosen because blower motors experience a much lower performance degradation than flat-style fans do when faced with high static pressure. A 12V DC fan was chosen because the Arduino voltage regulator can take up to 12V, which meant a single wallwart power supply for both the Arduino and the fan.

There are many blower fan options available from DigiKey but the majority are brushless. Brushless DC motor actually have circuitry inside which performs a timed power distribution, turning the coils on and off as the motor rotates. When a PWM signal is applied, which is also turning the power on and off at a certain frequency. The interaction of the two creates a sub-optimal effect that can burn out the fan and even worse it makes an annoying noise. The problem here is that when using a PWM, you really should be using a brushed motor or a brushless motor with a PWM input.

Neither of these really looked to be an option so I did what others before me had done to work around: add a capacitor to smooth the output voltage. I couldn’t find anyone who could tell me how to select the capacitor size; they all were more interested in telling me how I was “doing it wrong”. Experimentation showed me that if I put too big a capacitor, it could charge too quickly and allow the fan to run at 12V at all times. I tried to solve this with a current-limiting resistor, but didn’t have much luck so I eventually just worked down the Farads until the fan speed was controllable and most of the noise was gone.

Fan circuit with smoothing capacitor and freewheeling diode

Note that I’ve changed from Bob’s design in that I use an IRL510 instead of an IRF510. The IRL is a 5V logic-level part and the IRF is a “standard” part, which it seems is 10V. This part might also be overspecced, considering it will pass 100V/5.6A and I only need 12V/180mA. If I were going to try to make these for profit, this would probably be something I’d try to save a little money on. I’d also try to figure out some way to save on labor, because I’ve probably spent at least 100 hours on the HeaterMeter so far. This circuit could use some other refinement too, I think I’m getting feedback right around the point where the fan turns on that sometimes causes corruption in the LCD display. That’s not a good sign.

Something that took me a long time to find an answer on was the diode across the fan leads. I couldn’t figure out what it was there for, and it appeared to be in backward in other schematics if it was supposed to be there to prevent the 12V line from being hooked up backward. This is called the freewheeling, snubber, supressor, or flyback diode. When an inductive load, such as a fan or relay is hooked to a MOSFET or any switched power, you should include such a diode. Fans and relays are coils, so the current going through them sort of has momentum. When you switch the current off, it wants to keep flowing. The problem is that the silicon switch in the MOSFET is preventing that from happening, so what happens is that the coil builds up a large reverse voltage. In some cases up to 100x the incoming voltage. This can arc and fry a silicon switch, do bad things to batteries, or burn contacts in physical switches. The diode makes the current loop around until it dissipates, and will generate a reverse voltage roughly equal to the forward voltage of the diode (if my reading is accurate). A 1N4001, which will stop up to 50V, should be sufficient for this application but any 1N400x should work.

Here is my overall v3 schematic for the Arduino BBQ Grill Controler project codename HeaterMeter. I actually built a v2 schematic in the construction post that will follow this, but I am slowly morphing it to a v4 for reasons I will later describe. The file is an EAGLE 5.0 schematic and board layout file. I think I’ve got everything needed to view it, but I’ve never distributed a schematic so you may have just downloaded 43kB of useless 1s and 0s.
HeaterMeter Schematic v3

Foods cooked on the grill are at the top of my food chain. I am a main predator of Kingdom Animalia, Phylum Grilledatas. The problem with grilling is that the grill is outside, in a place where it is usually hot, far away from my beer, and entertainment-starved unless observing the negative gravitropism of stenotaphrum secundatum is your thing. Barbecue further aggravates the situation by requiring long cook times which may even necessitate losing sleep. Sure you can “barbecue” in an oven, “barbecue” cooked in an oven isn’t barbecue all and bears as much resemblance to the real deal as K-rab does to stone crab. There has to be smoke, and the easiest way I know to get smoke it to light something on fire.

Contrary to what Frankenstein says, fire isn’t always bad, but it can require a good bit of fiddling make it do what you want, sort of light plate spinning. It certainly is a lot easier than plate spinning and you can’t eat anything when you’re done spinning plates for 8 hours, so I’m not sure why I used that analogy. Anyway, I’ve tried the Alton Brown Electric Pot smoker concept, which produced lackluster results even after several attempts and equipment swaps. The initial solution was to step up to the big leagues and buy a Big Green Egg, the self-described “World’s Best Smoker and Grill!” (emphasis theirs).

Enter the Egg

The BGE works really well and allowed me to go hang out at World of Beer while cooking BBQ, where my concern that the food was being either cremated or being cooked entirely with solar heat was moderately low. Some jockeying around with vent positions to get the temperature right over the first 30 minutes is required. There is also notable apprehension when leaving the grill behind which declines markedly with each passing Saint Rogue Red or Dogfish Head Midas Touch. Returning to find the Egg 15-20 degrees on either side of the sweet spot isn’t unusual, but acceptable considering the alternative is to drink alone.

To further complicate things, I used to live in a 3 bedroom apartment.

The AGC Happiness Scale

There is nothing downstairs worth getting on the ground floor for. It isn’t until you get to around 50 on the AGC Happiness or “Agck Happ” scale that life starts being worth living. Running up and down two flights of stairs while drinking is a sure recipe for disaster; I could fall down and the food would burn by the time the coroner came to retrieve my corpse. Therefore I needed a way to minimize the amount of time I spent in the peach section of the scale and maximize the amount of time I spend in the… fuchsia area? The periwinkle area? We’ll just say I want to be “in the pink”. Science needed to be called upon.

Google led me to finding The BBQ Guru Digi-Q line of products and Rock’s Stoker. These served the purpose, but 250 American wing-wangs? That costs more than many of the grills they are designed to be controlled! Not mine, mind you, because the Big Green Egg costs roughly what its gold weight equivalent is. More research lead to a post by Bob Hruska about his microcontroller project. I’ve always been interested in embedded development, but have never spent the time to look into it because the barrier of entry seemed pretty high. Bob’s project seemed within reach for me, and did what I wanted. Why don’t I just tell you what I wanted.

Design Goals

This project had to meet certain requirements to lift it from a glorified thermometer to an actual time-saving, effort-reducing, beer-intake-facilitating device. My mother also would like it to increase my life expectancy rather than abate it, but chicks dig scars and I’m Argh baby.

• Fully autonomous grill control capable of maintaining a preset temperature of a charcoal grill for 12 hours. Hands-free operation, just like Ghost Ridin’ the Whip
• Standalone 802.11 wireless remote query ability. Having just a temperature sensing unit with probe wires is bad enough already. Having to piggyback a wireless router, network cable and second power supply onto a wired solution isn’t a manly solution. Coming out with a bag full of grill control parts would make me feel like the girl who dumps her giant-ass purse on the table spewing lipstick, gum wrappers, and spare change everywhere
• Grill-local digital temperature display. I don’t want to have to use a web browser to see what is going on.
• No supplementary control equipment. Most home automation equipment can not function by itself, relying on a base station to mediate control. This is along the lines of the standalone wireless deal, but it bears repeating that there should be just one box that can travel without needed a van full of support equipment
• No permanent grill modification. Scarring my Big Green Egg would be like Angelina Jolie getting a tattoo. Err, I mean another tattoo. Like on her face or something.
• Cost less than the commercial solutions. I want it Better, Faster, and Cheaper.
• Room for customization and personalization. Nobody is perfectly happy with anything, or not as happy as they could be if it had a racing stripe or some flames on it.

And thus begun the Great Grill Control Project. The Great Wireless Grill Control Project. The Great Wireless Grill Control Project 1: Electric Boogaloo. In my next post I’ll describe the components of the system and some of the technologies they provide.

More information on why I like it and why to choose it over other timing options follow after the break.

Why Not TTimer?

TTimer is Delphi’s wrapper to the windows message WM_TIMER. Immediately that means that our code has to be accessible from a message processing loop, which may be inconvenient in the standard “you can’t get there from here” manner. The nature of the event-driven model means you’ll have to maintain state while you wait for the timer to fire, when often it is just more convenient to keep the state in the current scope of your task. The separation of the timer’s start from the event of its completion also can have an impact on the readability of the code and make the operation unintuitive.

Windows timer messages also aren’t suited for timing beyond a few events. A high volume of timer messages can clog up your WinProc message queue, making your UI less responsive if you’re handling them in the main thread. WM_TIMER is also defined as a low-priority message, which means delivery is postponed until no higher priority messages are in your message queue. The delay between the expected firing of the event and when the WM_TIMER actually arrives also may make the timer unsuitable for your application due to its imprecision.

Finally, the duration of a timer is limited depending on your platform. Older platforms used to limit the interval to an unsigned 16-bit word, or about 65 seconds. For longer intervals on these platforms, you’ll still need to maintain your own minute count. This shouldn’t be an issue, as everything from XP up supports 31-bit (yes 31-bit) timers, but something to keep in mind if you’re still supporting legacy code on unsupported platforms.

Why Not TDateTime?

Yes, why wouldn’t you store times in something that has time right in the type name? TDateTimes are sometimes suited for timing operations. Their benefits both stem from their absolute format; the timeout is a calendar value that can be visualized, and your timeout can be set any time from now until after you’re dead and buried. Hopefully you won’t need to be reanimated to maintain your code.

The drawbacks to using TDateTimes also come from their format. For starters, TDateTimes are actually 64-bit double floats, and measuring short-period timeouts with fractional days becomes non-obvious. For example, a cached item that will be flushed out in 5 seconds expires in 0.00005787 days which isn’t evident by looking at that number. I end up seeing a lot of timeouts specified in what I call “full day notation”:

if (dtElapsed > 1.0f/24/60/60*5) then

Your timer code is also invariably going to be calling Now() to measure start and elapsed times, which means a call to the kernel’s GetLocalTime, which in turn needs to get to the hardware Real-Time Clock. Slow. Working the TDateTimes means you’re also doing floating point math, which I try to avoid whenever possible for its performance-robbing characteristic.

Sell Me on GetTickCount()

GetTickCount() returns the number of milliseconds since the system was started in an unsigned 32-bit integer. Timing things in milliseconds is extremely intuitive and readable. 1 second is 1000 milliseconds. Everybody loves round numbers, with the exception of the people who invented the system of measurement used in the United States. The ability to read and quickly comprehend what the value represents can not be understated.

Another advantage is that GetTickCount, being an integer operation which references only an internal kernel counter, makes it fast. On a Windows 7 Core i5, a single start and elapsed measurement is 25x faster than with Now(). On older hardware yet still-current operating systems the difference can be more dramatic– 100x faster is not uncommon.

While GetTickCount is fast and easy to read, it isn’t without its faults. The first immediately obvious condition is that a 32-bit integer can only hold 4 billion milliseconds, which means the counter rolls over every 47.91 days. This means that GetTickCount is not suited for time periods to exceed 47 days. I wouldn’t consider the use of GetTickCount for anything more than even a day though, finding TDateTimes more intuitive for that operation. The second issue is granularity. GetTickCount isn’t a millisecond-resolution timer as you’d expect. In supported Win32 operating systems, the resolution is 15-16 milliseconds. If anyone can explain why it is varies between 15 or 16ms, I’d love to hear it.

The rollover is by far the most important thing to take away from this. Your code must account for rollover and also remember that your timer values need to be unsigned.

var
iStartTime: integer;
dwStartTime: DWORD; // or Cardinal
...
  if (GetTickCount() - iStartTime > 1000)  // WRONG: Widened both operands
    MessageBox('Fails after $7FFFFFFF seconds');
  if (GetTickCount() - dwStartTime > 1000)  // RIGHT: Signedness match
    MessageBox('You did it right');

Delphi will try to help you out on the first case, emitting the warning “W1024: Combining signed and unsigned types – widened both operands”. However, because iStartTime can only go to 2 billion, after iStartTime is greater than 24.855 days this comparison will always evaluate to false and your timer will not fire again. Ever.

Another important thing to remember is that the proper way to deal with the rollover is using the pattern I used above. Always use Current – Start (comparison) Interval not Start + Interval (comparison) Current. This is part of the magic of unsigned rollover. Let’s look at some cases with a single byte integer

// Case 1 - Current: $00   Start: $FE  Interval: $0F
Current - Start > Interval
  $00 - $FE = $FFFFFF02
  // ^^ only the bottom 8 bits are retained leaving the correct answer, 2
  // Still below Interval

// Case 1 - Current: $00   Start: $FE  Interval: $0F
Start + Interval > Current
  $FE + $0F = $0D
  // ^^ $0D is less than Current, so is true after 2 milliseconds instead of the 10
  // Erroneously returns true for from MAX_INT-Interval to MAX_INT 

// Case 2 - Current: $0D   Start: $02  Interval: $0F
Current - Start > Interval
  $0D - $02 = $0B
  // ^^ Simple math, no rollover

// Case 2 - Current: $0D   Start: $02  Interval: $0F
Current - Interval > Start
  $0A - $0F = $FE
  // ^^ Looks like a ton of time has passed
  // Erroneously returns true from $00 to Interval

Summary

• Rollover is 49.71 days Do not attempt to time intervals of this duration.
• Storage type must be unsigned Use the DWORD or Cardinal data type to avoid automatic operand widening.
• Compare as Current – Start (compare) Interval Other patterns fail on the corner cases.

So there you have it. GetTickCount() is a high performance timing method that is easily understandable as long as you understand the rollover and use the right types.

Consider the following code analog to this situation, where the Delete button looks too much like, or is placed to close to the Save button in a toolbar. As a defensive programmer you safeguard the delete function with a confirmation:

  if MessageDlg('This action is not undoable.'#13 +
    'Are you sure you want to delete all your work?',
    mtConfirmation, [mbYes,mbNo], 0) = mrNo then
    exit;

  DoDeleteAllYourWork();

This works fine and the early bailout situation is preferable in my book when the code of DoDeleteAllYourWork() actually follows inline. The code is written, compiled, shipped and you forget all about it until a few years later someone calls up spitting venom because they pressed Escape and your app deleted their work. What has occurred is a subtle issue created by Delphi switching from their own MessageDlg to Vista’s powerful TaskDialogIndirect() API when you enable runtime themes in your project options.

The little check with 100 implications

So what does Enable Runtime Themes actually do?
Checking this box actually tells Delphi to compile a manifest into your application. A manifest is a text XML resource of type RT_MANIFEST (24), which specifies the activation context of your application. In the case of the enable runtime themes and Delphi, this creates a CREATEPROCESS_MANIFEST_RESOURCE_ID (1) type manifest to request a version 6 COMCTL32.dll– the version that supports XP-style theming. This has a few implications (some dependent on windows version)

• All your controls coordinate better (“theme”) with the desktop.
• Teaches you what the ParentBackground property means.
• Directory and registry virtualization is disabled, meaning your shitty code writing to HKEY_LOCAL_MACHINE, %ProgramFiles%, etc will stop silently writing to %APPDATA%\Local\VirtualStore and start failing.
• Alters the behavior of of MessageDlg.

How does MessageDlg differ with theming enabled on Vista and above?
Delphi used to build the dialog created by MessageDlg itself, actually constructing a TForm descendant called TMessageForm and programmatically creating the buttons and labels on it. Microsoft however included a powerful API starting with Vista that allows you to create simple themed Yes/No or selection dialogs with very little code. Once you turn on theming, Delphi will start using these style dialogs on Vista (Windows Version 6) and above instead of creating its own TMessageForm classes.

  // Excerpt from MessageDlg in Delphi 2007
  if (Win32MajorVersion >= 6) and UseLatestCommonDialogs and ThemeServices.ThemesEnabled then
    Result := DoTaskMessageDlgPosHelp('', Msg, DlgType, Buttons,
      HelpCtx, X, Y, HelpFileName, DefaultButton)

The difference is, and here’s where your tea-desiring patron gets stabbed in the gut, they behave slightly differently with regards to what is returned when the user cancels. The code at the beginning of this article works as expected when running with theming turned off, but starts deleting data when theming is turned on. What’s happening is Windows returns IDCANCEL if “Cancel button was selected, Alt-F4 was pressed, Escape was pressed or the user clicked on the close window button.“. This behavior can occur regardless of if there is a cancel button on the dialog. The simple solution is to just test for the mrYes result or check for both negative options

  if MessageDlg('This action is not undoable.'#13 +
    'Are you sure you want to delete all your work?',
    mtConfirmation, [mbYes,mbNo], 0) in [mrCancel,mrNo] then
    exit;

  DoDeleteAllYourWork();

This is a rather simple problem with an extremely simple solution, but one of many gotchas that comes with requesting theming. If you’re interested in knowing more about TaskDialogIndirect(), check out the Delphi TTaskDialog class, or the simple TaskMessageDlg() which is a MessageDlg with an better looking title line. Welcome to the world of ::MessageBox() 2.0!

{
uses ActiveX;
Ported from John Robbins - Microsoft Systems Journal Bugslayer Column - October '99

http://www.microsoft.com/msj/1099/bugslayer/bugslayer1099.aspx

}
function DebugCoGetThreadingModel : integer;
const
  OLE_APT_MASK  = $0080;
  OLE_FREE_MASK = $0140;
var
  dwOLETLS: Cardinal;
  dwFlags:  Cardinal;
begin

  asm
    // Get TEB
    mov eax, FS:[018h]
    mov eax, [eax+0f80h]
    mov dwOLETLS, eax
  end;

  { Not initialized }
  if dwOLETLS = 0 then begin
    Result := -1;
    exit;
  end;

  dwFlags := PCardinal((dwOLETLS + $0C))^;

  if ((dwFlags and OLE_APT_MASK) = OLE_APT_MASK) then
    Result := COINIT_APARTMENTTHREADED

  else if ((dwFlags and OLE_FREE_MASK) = OLE_FREE_MASK) then
    Result := COINIT_MULTITHREADED

  { Unknown }
  else
    Result := -2;
end;

And some sample usage code

  case DebugCoGetThreadingModel of
    -2: Label1.Caption := 'Unknown';
    -1: Label1.Caption := 'Not initialized';
    COINIT_APARTMENTTHREADED: Label1.Caption := 'COINIT_APARTMENTTHREADED';
    COINIT_MULTITHREADED: Label1.Caption := 'COINIT_MULTITHREADED';
  end;

John also has an excellent blog of his own which is which is a fantastic low-level technical resource. Speaking of old magazines, I just discovered the Bug of the Month ad that has appeared in Dr Dobbs for nearly the past 20 years has an online archive. Head over to Gimpel Software if you’re up for some C-based brain exercises.

• Your application is crashing remotely The worst place for an application to fail is when there is no debugger available. Usually these problems can be corrected via the “Works Fine Here” solution. Unfortunately most bug-tracking systems lack this option to close tickets.
• Lack of symbol information The best trace you can get in the debugger shows a thread balls deep into Windows API calls for no apparent reason. Some resolution can be gained from the DLL export addresses but without proper symbol information this isn’t enough detail.

The solution to both these scenarios is to use Microsoft’s free debugger WinDbg, available as part of the Debugging Tools for Windows package. You’ll need the version for the host platform where the trace will occur. This means get the 64-bit if you’re debugging on Win64 even if your app is 32-bit. You’ll also need the map2dbg utility, originally by Lucian Wischik.

Prepare Your Project

First, you’ll need to prepare your project so that it exports its symbols to windbg, which is going to run externally from Delphi. To do this you need to change the Linker options to generate a map file with a list of Publics. Compile your app and run map2dbg against the exe to generate a .dbg file. You can have Delphi automatically do this for you adding a Post-build step to the project options. If you’re debugging a 32-bit application under a 64-bit environment however, windbg expects the dbg file to be named image00000000`00400000.dbg so that can be added as well. DLL .dbg files can stay the same as the module name.

C:\SysTools\map2dbg.exe "$(OUTPUTPATH)"
move "$(OUTPUTDIR)$(OUTPUTNAME).dbg" "image00000000`00400000.dbg"

Delphi Project Options

Using WinDbg

Fire up windbg and launch your application using File | Open Executable…. Now set your Symbol Path from the File menu as well. Start with the directory where your application’s .dbg file is located, then any DLL .dbg files, and finish with “srv*”. This special token instructs WinDbg to use the Microsoft symbol server, which will automatically download the symbols you need and cache them in %ProgramData%\sym\dbg.

Symbol Search Path

0:008:x86> !sym noisy
noisy mode - symbol prompts on
0:008:x86> .reload

The hotkeys are standard Microsoft versions so that means F5 means run, or type ‘g’ into the command window. For this example I’ll be debugging a 32-bit lockup on my 64-bit development system. Once I’ve got my application wedged, hit the pause button. If your development environment is the same as mine, the first thing you’ll notice is that all your threads are in the wow64cpu dll:

0:001> k
Child-SP          RetAddr           Call Site
00000000`0069f0f8 00000000`74db282c wow64cpu!CpupSyscallStub+0x9
00000000`0069f100 00000000`74e2d07e wow64cpu!WaitForMultipleObjects32+0x32
00000000`0069f1c0 00000000`74e2c549 wow64!RunCpuSimulation+0xa
00000000`0069f210 00000000`773ed177 wow64!Wow64LdrpInitialize+0x429
00000000`0069f760 00000000`773a308e ntdll! ?? ::FNODOBFM::`string'+0x2bfe4
00000000`0069f7d0 00000000`00000000 ntdll!LdrInitializeThunk+0xe

This is because windbg always shows the architecture of the context where the break occurred. If you hit Pause, you get 64-bit. If a breakpoint in your code pops, you get 32-bit. Luckily, the you can toggle architectures using the wow64exts extension by typing !wow64exts.sw, or simply !sw.

0:008:x86> k
ChildEBP RetAddr
04daf56c 770b0816 ntdll32!NtWaitForSingleObject+0x15
04daf5d8 76211184 KERNELBASE!WaitForSingleObjectEx+0x98
04daf5f0 76211138 kernel32!WaitForSingleObjectExImplementation+0x75
04daf604 763188df kernel32!WaitForSingleObject+0x12
04daf628 7643a819 ole32!GetToSTA+0xad
04daf658 7643c05f ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x140
04daf738 7632d0e5 ole32!CRpcChannelBuffer::SendReceive2+0xef
...

Ah that’s better. Now we see a common problem, a COM object is trying to call into an STA and when something hangs like this, it is generally because the STA has stopped pumping messages (calls to STAs are serialized using windows messages). Let’s do a quick check to see what the threading model of this blocked thread is, which requires a little inside knowledge of how the windows keeps track of OLE threads. When you call CoInitialize()/CoInitializeEx() for the first time on a thread, the threading model you request is stored in the thread’s Thread Environment Block or TEB. Sidenote: Windows95 called this the Thread Information Block (TIB), and you’ll sometimes see the terms used interchangably although some of the data contained in the block differed between the 95 and NT platforms. This structure is allocated by COleTls::TLSAllocData and is stored at offset 0xf80 in the TEB, and the offset of the threading flags is at 0x0c in the structure. From this knowledge, I can use !wow64exts.info to get the TEB address, add 0xf80, dereference that pointer then inspect the flags to determine the threading model. Note that I use 0x notation for hexadecimal values for clarity, but windbg always takes numbers to be hex anyway so the 0x is optional.

0:008:x86> !wow64exts.info
...
TEB32: 0x7efa0000
0:008:x86> dd 0x7efa0000+0xf80 L1
7efa0f80  006cf638
0:008:x86> dd 0x6cf638+0x0c L1
006cf644  00000143

Using some more Win32 knowledge, I happen to know that the TEB is also pointed to by the Frame Segment pointer. I can use this information, the POI() debugger macro, and my ability to count to 0x0c to reduce this operation to a single quick command.

0:008:x86> dd poi(fs:f80)
006cf638  00000000 00000000 00000000 00000143

In these flags, if the bits 0×140 are present, the thread is in the MTA. If the bits 0×080 are present, the thread is an STA. If you get ????????, this means that CoInitialize() has not been called. The flags here are 0×143, indicating MTA. So our thread here is MTA, trying to call an STA. Lets look further into the backtrace to see which thread this is and where my code ends and OLE32 begins.

0:008:x86> k
ChildEBP RetAddr
...
04daff08 00510119 ScriptDPC!ScriptHost.TScriptHostImpl.Unlock+0x6
04daff30 0054a986 image00000000_00400000!DPCHandler.TDPCHandler.DoShutdown+0x35
04daff74 004055e6 image00000000_00400000!MsgConBase.TMsgConBase.ActionShutdown+0x36
04daff88 76213677 image00000000_00400000!System.ThreadWrapper+0x2a

The last bits of my code are in the ScriptDPC dll, so I’ll set a breakpoint in the Unlock method to step through and see what’s going on.

0:008:x86> bp ScriptDPC!ScriptHost.TScriptHostImpl.Unlock
Syntax:         Module     Unit         Class      Method

Use bp to create a breakpoint, bl to list them, and bc to remove (clear) one. Single stepping through the function shows that while clearing up the script context, global COM objects referenced by the script are getting released. Hrm, so it seems one of our COM objects is being Release()d by this thread but it was created by an STA so OLE is being nice and trying to RPC to that thread to release the interface. All inter-apartment calls are considered RPC regardless of their activation context, be they in-process, out-of-process, or on another machine activated by DCOM/COM+. Looking through all of my threads using the method above, I see only one is an STA, the main thread. So what I am probably looking for is the main thread creating a COM object in my script and not releasing it, which gets deferred until the script context is deleted.

To find where this occurs is a bit more tricky, so I examine the points where the UI might interface to my scripting component. While poking around examining the interactions, I notice this DLL load trace coming up in the debugger when I execute a context menu a certain component:

C:\Windows\SysWOW64\RpcRtRemote.dll
c:\IE\DPCs\Test\DBServices.dll

The first line there indicates that the RPC engine is being loaded into my process. This creates an MTA thread and a pool of worker threads to service method invokations for objects and is a dead giveaway that the UI’s STA thread has just created a free-threaded object, the calls to which must be marshalled cross-apartment. Some quick tracing through the source code turned up this bit of code being called from the main thread:

procedure TScriptDPCBase.ScriptScriptObjectsNeeded(Sender: TObject);
begin
  RegisterScriptObject('IEProxy', IEProxy);

  if not Assigned(FDBServices) then
    FDBServices := CoCDataBaseServices.Create;
  RegisterScriptObject('DBServices', FDBServices);
end;

CDataBaseServices is flagged as tmFree. There’s our MTA object. A little rearranging moved the CDataBaseServices creation to the initialization of the script thread, make sure it isn’t being accessed by the UI, and lockup solved.

• Hopefully share knowledge with someone else who has run into the same problem I have.
• Provide a repository where I can remember the solutions to problems I’ve had, because I might have to use them again.

Let’s start with some information about who I am.

My name is Bryan Mayland and I am a programmer for a small software shop in Tampa named Leonine Development Services, Inc. I’ve been coding professionally now for 15 years or so, from back in the 16-bit windows days. My primary language of choice is Delphi for the win32 target, but I have a pretty strong background in C/C++, can get around pretty well in Intel x86 assembler and C#.NET, and a rudimentary understanding of PHP and LUA.

What the hell is Delphi?
Delphi is a product created back around 1995 to compete with Visual Basic by Borland (which did a corporate makeover and became Inprise, which spun back off their development tools division as Borland, which became CodeGear, which was bought by Embarcadero). Delphi 1 was a 16-bit Object Pascal compiler with Rapid Application Development (RAD) features. This meant you could design your GUI in a GUI in the same way you could in Visual Basic except the end result would not be a completely shitty interpreted app whose array indexes start at 1. Seriously, everyone knows the universal first number is 0.

Coming from a 16-bit straight C background, Delphi was a godsend. The programs compiled and linked ultra fast, you could throw together a decent GUI in a heartbeat, and the resultant native code was pretty quick too. Of course, it was also a 16-bit “joy” when a unit would compile to more than 64k and the compiler would choke. Same story for when some dufus declares a bunch of initialized constant shortstrings and you get the dreaded “Data Segment too Large” error. Delphi 1′s .data segment was shared with the stack segment, so the trick there was to lower the size of the stack to fit your data, but not so small that you’d run out of stack space while the program was running.

Suffice it to say, the Delphi of today has come a long way since then, and it is even more impressive when you look under the hood. That is why I’ve started this blog, to share insights as to what is under the hood in everything from Adultery to Zoology.

I googled your name and all I found was personal listings on Craigslist
For some reason those keep bubbling up to the top, but the guys who call are always nice so I don’t mind. I have done more noteworthy things, I assure you.

• Gnutella I reverse engineered and published the gnutella protocol, helping to start the glorious age of Peer to Peer topologies.
• DAOCSkilla reverse engineered the network protocol for Dark Age of Camelot and wrote a radar program so I’d never wander around East Svealand again. This made a lot of people upset for some reason, but it also led to this great list of monster locations as well as a security advisory when I saw that the credit card authorization system used incredibly weak passwords, and another when they still didn’t get it right.
• XBMC I’ve contributed to the open source media center application XBMC. I was responsible for the flags that indicate the resolution of videos and the codec of the audio streams, as well as some other small fixes. I’d love to work on it again if I ever find the time. The developers are extremely talented and are my kind of jerks.
• Linux I may still have one or two lines of code in the Linux kernel. Debugged an kernel issue with VESA framebuffers.
• XViD Wrote the first multi-threaded I-frame motion search algorithm, boosting performance of the video encoder by up to 60% on a dual-processor/core system with identical output. XViD currently uses a different algorithm now that works on all frame types I believe.
• Wrote a service for a DOT COM company that processed, stored, and indexed every stock tick for every trade on the US markets in real time. On a Pentium II 300. A friend and I started working on it one Saturday morning, and it replaced the production system on Tuesday. Then I drank a lot of beer and slept for a day.
• I’ve also written a Meditech MAGIC terminal emulator by reverse engineering the protocol (see a trend developing here?). That may not sound like a very big accomplishment, but damn I’ve spent some long nights on that.
• I fancied myself as a game programmer for a while, although I use the term loosely as my services were never credit-worthy, and in some cases not even payment worthy in the case of one MMOG.

So there’s my start. I think I’ve gotten past all the boring backstory and now I start recording actual important things in this blog. I’ll start it with this:
Screw you, John Lam.