Elementary school programming

July 30, 2014

An embedded.fm listener (Mike) is working on starting a program for a local elementary school that introduces Engineering with a main focus on programming and electronics. He wanted to know if I had any suggestions.

For programming for almost any age, Scratch is awesome. It can be web based or on a Raspberry Pi. It is very well design to introduce programming concepts (and it is fun to make little movies of avatars). I’ve used it for 2nd graders (and 11th graders).
For 3rd-6th grade, Minecraft is a good way to get some kids interested in programming, they end up wanting to run their own servers and create worlds. (Though it tends to be a bit boy-centric. My godsons have adored it since they were in 2nd and 4th grade (now they are 5th and 7th)).
For electronics, the LightUp system is pretty intuitive and very neat to play with. They have kindergarten stories. The kits are a bit expensive and they are only from MakerShed (big kits aren’t available yet).
In the meantime, I have a friend who picks up Snap Circuits at garages sales. His 5 year old is building amazing things (making mods on the existing plans already!). I haven’t played with those but am amused by the pics I’ve seen.
So what would you suggest?

Email: NaNoWriMo

July 27, 2014

Friday at lunch, I mentioned NaNoWriMo (national novel writing month). It isn’t until November but I admit I sketched out my outline starting in August.


They do the “contest” as a fundraiser, raising money for libraries in distant lands. It is free though they ask for donations. They supply support, forums, local meetups, and some prizes if you finish (one was a radio station that would let you read a chapter, another was coupons for getting books printed at a self publishing site).

It was fun. More importantly, it made me confident I could write a book when the time came that I wanted to do a real one.



My checklist for debugging insurmountable issues

June 21, 2014

I listen to the Amp Hour podcast. Last week, they were talking about ways engineers fail and about checklists as a way to be more disciplined in avoiding preventable errors.

I try to learn from my mistakes so I do have a checklist of sorts when I reach the “it’s all broken and never, ever going to work again” stage of debugging. Of course, when I get to that point, the symptoms are usually different. Still, some parts of the path are relatively common.

1. Having you tried turning it off and back on again?

It is a joke, a very well seasoned joke. But it is only funny because it is so horribly true. This isn’t a good way to debug a horrible problem but it does provide the circumstances the problem happens under. I often start from scratch to reproduce issues because there is often a clue in the process. So, turning it off and back on again is a way to make sure that I start from a well-understood starting point.

The next questions:

Really? Are you sure?

But these are going to follow most of these checklist questions. As embedded system get more complicated, it is pretty easy to turn a part of the system off but not all of it. I don’t usually go from no-power-to-my-desk, I almost always start with my computer on. But sometimes, turning off all power is necessary (including restarting the debugger).

2. Does it have power?

This is different than the previous in that the “it” refers to all the things that may be broken: the processor, the debugger, the sensor or actuator, the level shifters, everything. This step usually requires a voltmeter (and is related to #5).

3. Is it running the code you think it is?

I sometimes have two or three code bases. If my development environment is pointed to the wrong one, I could easily be compiling over here but loading the image from over there. Or possibly, if the load process is difficult, I may think I’ve loaded the code and somehow missed a step.

I’m working on a big system now, a monitor of another system. When I compile the code, it goes through four machines before it gets to where I can try it out. The possibility that I typed cp instead of scp (or forgot the ending colon!), well, let’s just say it happens more often than I’d like.

This is the reason to have build numbers. But if you are really, really sure it is running the code you think, change something about the output and reload. Make sure you see the change.

4. Did you read the boot and debug output?

I write error logging features and always want a serial debug output. I love power on self tests. However, once the system is working, I stop looking at them. But sometimes, if a cable has loosened or hardware has failed, the system will tell me. But only if I am listening.

5. If it used to work, what changed?

Nothing changed, of course. It just stopped working. My minor code change could not possibly have caused something so catastrophic.

I’ve heard that. I’ve said it. That doesn’t make it true.

If nothing changed, then run the old code. If it fails, well, that’s interesting now, isn’t it? If it succeeds, well then, stop saying nothing changed, something obviously did.

This does require frequent commit to version control, to get back to a last-known-good image. But you were doing that anyway, right? And then you can binary search to determine where the error crept in.

Note that if the code change doesn’t explain it, compare the map files (even the binaries). Realizing that something crossed a boundary may give you inside. Oh, also, the makefile (or project file) may have changed: optimizations can have big ramifications.

6. Is there anything interesting in the map file?

Like many embedded engineers, I find the map file to be a bit of an illegible mess. Happily, I’m not afraid of them anymore. There is an amazing amount of information in the map file. And it provides a different perspective on the code, sometimes it will jog my memory.

7. Can you prove it is hardware?

Of course, at this point, it probably is. But you know hardware engineers, they can be feeble. They need proof. So what kind of proof can you offer? How can you break it down to show it cannot possibly be the software?

Seriously, I have had the privilege of working with some phenomenal hardware engineers. It is seldom hardware (but not never). The process of proving it is hardware is a good part of debugging. Plus, if you can make the difficult error repeatable for the hardware engineer, they’ll probably take you to lunch for making their jobs easier.

8. Can you explain the problem to another engineer?

When I tutored intro to CS, I asked people to explain their problem to a teddy bear outside my office before explaining them to me. I usually listened in. However, at least 50% of the people thanked the bear and left without talking to me. Ok, probably only 20% thanked the bear but most walked away because they never actually needed my help. They needed to get their thoughts in order, to explain it to themselves.

I do that now, talk to myself. Sometimes I try to explain it to a trusted colleague (or a junior engineer) in email, trying to figure out what questions they would ask me so the explanations is really good.  Occasionally, I even send the email after all that, if I still can’t figure out the problem.

9. Did you use the single line if again?

When I saw Apple’s goto Fail bug, I completely understood.  I avoid unbraced if statements because one month, I tallied up my most common coding mistakes and found that unbraced if statements caused a disproportionate number of my bugs. I vowed never to use them again.  Since this is a known failure point on my part, it makes my checklist.



Scribbles to myself

May 28, 2014

I did Unix system administration in college. That was many, many years ago. And really, I managed the consultants, wrote quick references for users, made new accounts, and only filled in on deep technical management when someone made me (usually when someone else was sick or had lit something on fire). Good times. But I really was an expert unix user at one time for multiple unix varieties (hey, the math cluster was hpux so I got deep into that the summer I spent working on computational math libraries to model fluid flow).

But, as I mentioned, it was many, many years ago. Since then, I’ve played with Linux, dabbled here and there. I’m more comfortable with Mac OS in the command line (yes, I’m that awful person who remapped my flower and control keys so my fingers didn’t need to re-learn ctrl-z when developing with xcode).

My next contract will be all Linux-y and I’ve been wanting to do more embedded Linux (why is everyone so excited? When I played with it in 2006 it seemed like a great way to spend $100k in development time and then switch to something deterministic). Having borrowed a Beagle Bone Black, reinstalled Windows to have 64-bits so I can access all of my RAM, and installed a virtual machine so my husband will stop laughing at me when I destroy things, I’m ready.

My first mini-project is to rebuild the BBB’s Angstrom distribution. The board I have is a bit old and the OS has been updated. It would be nice to return it from whence I borrowed it, all updated.

Of course, it isn’t that easy, I’m plagued with stupid things I feel like I should know. And I’m reading Chris Hallinan’s Embedded Linux Primer: A Practical Real-World Approach (2nd Edition). I read the first edition many years ago (about the time it came out since we were still working on the embedded Linux project then, though my role was manager-only, not developer).

As I struggle with getting everything set up and configured as I like, I figured I should note some of my favorite commands.

On my Linux VM, here are some of the things I shouldn’t forget:

> cat /proc/version 
Linux version 3.8.13-16.2.2.el6uek.x86_64 (mockbuild@ca-build44.us.oracle.com) (gcc version 
4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Tue Nov 26 08:41:44 PST 2013

I like to know where I am and what I’m running.

> sudo usermod -a -G dialout elecia

Since I got an off-the-shelf VM (Oracle’s Linux 6), I wanted an account for myself. I know better than to run as root all the time. On the other hand, I keep failing to be able to use things. It took me a stupidly long time to remember that I have to log out after changing permissions.

> screen /dev/ttyUSB0 115200

As long as I remember to switch my USB-serial cable to the VM, I can snoop on the BBB as it boots up and use a command line there. U-boot is neat. Also, control-a then k kills the screen but leaves it openable (it detaches).

On my unmodified BeagleBone Black

root@beaglebone:~# cat /proc/version
Linux version 3.8.6 (koen@rrMBP) (gcc version 4.7.3 20130205 (prerelease) (Linaro GCC 4.7-2013.02-01)
 ) #1 SMP Sat Apr 13 09:10:52 CEST 2013

I think I’m going to need to learn more about Linaro. I has come up a few times. I don’t think it is like ucLinux which can run without an MMM. Still, Linaro seems common for microcontrollers (the system-on-a-chip (SOCs)) that I’m likely to want to use.

This one is general more general:

ln -s /usr/local/bin/python2.7 python

That is linking the source (/usr/local/bin/python2.7) to the dest (python in the current directory). I’ve missed symbolic links.

Alias is ok:

alias ll="ls -lags"

But it should be noted that friends do not do

alias vi="rm -rf"

When their terminal is left open in a public environment. That’s just wrong. Of course, the way I use vi, it might as well be right.

I’ve been trying to stay in the Linux environment for most of the stuff I’m doing. When I find myself typing in questions into my Windows browser, I stop and go back to Linux. The fact that the VM captures my keyboard so I can’t alt-tab out of there is probably a good thing.

Building Angstrom has been difficult, lots of dependencies that are required to be something else by another part of my OS. Since I got the off-the-shelf VM from Oracle, I don’t think I got what would have been easiest to build Angstrom. I’m not sure what they want but it isn’t the Linux I have.

Oh, another good command to remember:

 find . -name sanity.conf

Where the dot is where to search (from here through all directories) and -name is what to search for. I initially typed “find sanity.conf” which leads to a pleasing error message

find: `sanity.conf': No such file or directory

But that was the clue to me that find was one of those trickier commands that I usually mapped.

I was looking for sanity.conf because I got error messages that suggested I look in there for more information. Inside, there was a comment:

# Expert users can confirm their sanity with "touch conf/sanity.conf"

Windows doesn’t have this whimsy. I’d forgotten that. Linux is more clearly built by people, with all of the interpersonal spats associated therein.

Windows is more of a machine, with the personality of an annoying, talking paperclip. Linux is more of a crowded bar with lots of people talking in many groups. If you can find a quiet spot and a good group, you’ve won.

But standing on the outside, trying to figure out where to go, is excruciating. And wearing the wrong distribution can get you shunned.




Are you ok is up!

May 23, 2014

The build instructions for my are-you-ok widget is up on SparkFun! How neat!

This isn’t the end for that project. I’ve been working on getting email on Maxwell (surprisingly not difficult). I’m going to visit Hugh this weekend to see why his accelerometer is fussy.

In a couple weeks, Elizabeth will be on the podcast again to talk about what needs to happen next, if she’s happy with the system and what changes to make. There is a rumor that SparkFun will have a kit of parts for me to give away at that time to podcast listeners. (I need a contest! Guess a number? The quotes are too easy thanks to google.)

For someone who seems to be always starting a contract next week (sigh), I have been busy. My EELive talk is going up on element14. I’m staying about a week ahead though this week I have to do the summary and I’m not ready.  Also, on element14, Sophi Kravitz asked me questions about consulting but I kept distracting her with RTOSs and stories of are-you-ok widgets. I’m happy with her resulting interview.

I went to the SOLID conference. it was interesting and eclectic. O’Reilly gave a big stack of my books away while I signed them and stress-chatted with people. I was there as press, recording things for the podcast. But Christopher says the noise level is too high, the results are too difficult to listen to. Argh. I’m not sure what I’m going to do. I do need to do something to “pay” for the press pass (and all the people I talked to).

I am still working with the Beagle Bone Black, though slowly. I updated my MacBook Pro from Win7 32-bit to Win7 64-bit which lets me use more than 2G of RAM. That is a complete re-install so I’m still finding things I forgot to back up (my bookmarks!). One reason to do this was to run virtual machines so now I have Linux running too. (It is the Oracle 6 one which seems to be Fedora based, I’m still orienting on how things work.) I’m trying to build Angstrom, just to update the OS that is on the board (step 1: update with known good image, step 2: update with my built image that should be the same as the known good, step 3: break everything).

I’m currently installing Python 2.7 because some precursor to actually building Angstrom needs it. It just gave me an error in the build process of python because it doesn’t have some library it needs. This is exactly how I remember Linux being.

I hope you have a good, relaxing weekend. I plan to.