Will real-time data processing replace batch processing?
At Confluent's user conference, Kafka co-creator Jay Kreps argued that stream processing would eventually supplant traditional methods of batch processing altogether.
Absolutely: Businesses operate in real-time and are looking to move their IT systems to real-time capabilities.
Eventually: Enterprises will adopt technology slowly, so batch processing will be around for several more years.
No way: Stream processing is a niche, and there will always be cases where batch processing is the only option.
Edge Computing / Software Development

Development: Git for Small Physical Computing Projects

Dec 10th, 2021 8:43am by
Featued image for: Development: Git for Small Physical Computing Projects

The first few articles of this git how-to series covered the basics of installing, populating and interacting with git repositories. Linux is a natural environment for git because the commands are easily entered from a regular text terminal at the command line. Although you initially use an internet-connected browser to create a GitHub account, for remote storage and sharing, pushing and pulling data with the command line is pretty straightforward. Accessing the content is easy whether local or up in the cloud.

Git was designed to organize raw, text-based source code and works very for that purpose. Can it effectively work with other content? What are the logistics of handling projects with a mixture of file types under git? Are there any basic recommendations for getting started?

Let’s explore these ideas using one of my Steampunk Arduino projects, as an example.

Programs for My Physical Computing Projects

esp8266-tft project board

esp8266-tft project board

Physical computing projects require a combination of applications for various purposes and produce a variety of file types.

All the programs I use, run under Linux and are freely available on the web. Likewise, Linux is available in dozens of flavors, to suit every type of computing hardware and personal preference. I like the Xubuntu 64-bit variant and run it on a two-year-old Dell notebook with 16 GB of RAM, a speedy 500 GB SSD, NVidia graphics and an Intel I7 processor. Multiple applications easily run simultaneously, without ever even turning on the fan.

The Arduino IDE (interactive development environment) is used to code and upload firmware to an Arduino micro-controller board. Similarly, the Processing language is for building little screen-based Linux applications that interact with Arduino or Raspberry Pi hardware. Both Arduino and Processing are feature-rich, easy-to-use and well supported for prototyping projects.

The libraries are particularly helpful and cover a lot of cool physical computing needs. With the board manager, the Arduino IDE can output firmware for dozens of different “Arduino” types of hardware, including various ARM-based micro-controller boards, ESP8266 wifi modules, the ESP32 NodeMCU wifi devices and more.

Gadgets need appropriate packaging for demonstrations, display and everyday hands-on use. 3D printing is a great solution. A nice enclosure gives a certain degree of respectability as a “finished” product. My steampunk projects often also include gears, mechanical links, brackets, nameplates, adapters and such. It makes sense to keep those 3D designs all together with the rest of the project files.

I’ve spent the last year getting up to speed using FreeCad for 3D model design. It features parametric modeling and spits out object files that go into the Prusa Slicer slicer program in preparation of printing on the Prusa MK3S printer. FreeCad CAD models (.FCStd1 extension) will easily reside in the git project directory. I don’t bother putting the object files (.obj extension) under git control though, because they change so frequently there really isn’t any point.

I also don’t include the g-code files (.gcode extension) that handle printing on the MK3S. Perhaps when I reorganize my gadget production operation and have a decent inventory of standard parts those files might be something to include. The FreeCad files are updated over time as the design evolves, so putting them under git control is reasonable. The CAD file size of my current model development is usually less than 3 MB per part.

FreeCad and the Prusa MK3S are the perfect combination for designing and physically producing interesting Steampunk gadgets and parts.

I use Fritzing, another free application, for capturing schematics and wiring diagrams. It is easy to use and well supported. It outputs nice, clean drawings for use in articles in the form of .png files, so they are fairly small in size.

My example esp8622-mqtt-lcd project also includes a few hardware graphics and screenshot files. These tend to not change much and probably can just reside in the project directory. You may or may not want to push them up to the GitHub repo.

Projects might also include any number of the following types of information.

  • Build instructions
  • CAD models for parts, enclosures, etc.
  • Documentation, usage, requirements, 3D printer settings, etc.
  • Idea maps
  • User interface pictures
  • Videos
  • Issues
  • Version index, cross-references
  • Parts list, part sources, Bill-of-Materials
  • List of useful links

Local 8266-tft directory listing

More Detail on the Arduino and Processing Text Files

Arduino code is the source of the firmware that runs on an Arduino micro-controller.

Lately, I’ve simply dumped each set of project files into an appropriately named subdirectory under /home/rob/parts. For example, my esp8266 nodemcu microcontroller with a 2.0-inch color LCD screen gadget for displaying MQTT messages is at /home/rob/parts/esp8266-tft. You can use whatever combination of directories works. Make sure to remember to go to that directory when you initialize that particular git instance. Likewise, I use a similar “directory”/repo naming up on GitHub and GitLab.

arduino sample code

Arduino sample code

Arduino and Processing code are very similar and look much like the C language. The source code are both text files and usually very small. Arduino files are compiled into binary files that are uploaded as firmware to the Arduino hardware. Processing files normally are executed from the Processing IDE, although they can also be compiled into standalone binary packages that will execute at the command line and outside of the IDE. I don’t normally retain the Arduino and Processing binaries in the project directory.

Both types of files are also stored in “sketchbooks.” Arduino files use a .ino extension, while Processing files use a .pde extension. You can have any number of sketchbook directories, that are then referenced from each IDE. To retrieve the files for editing, uploading and executing you simply roll down under the file menu, select “sketchbook” and click on your project name. Both IDEs set their “sketchbook” from a “preferences.txt” file.

Alternately, you might simply forgo using sketchbooks in the IDEs and just save the file into the git directory you are using for your project. That source code file, whether Arduino or Processing would then be called up using the “open” or “recent” menu item under the “file” drop-down, on the respective IDE.

Another point is that the Arduino and Processing files have to be stored in their own directories. git handles directories just as well as text files, so there is no problem in this regard. Pushing these subdirectories to GitHub also works without issue.

A Word about File Sizes

You may have noticed I mentioned the general output files sizes for the various applications. While it doesn’t make much difference if you just use git on your Linux notebook, putting files up on a git service like GitHub or GitLab is a little trickier.

github 8266-tft repo

github 8266-tft repo

GitHub for example has free and paid accounts. The free account allows 500 MB of space, on their servers for your projects. The tier-1 paid account allows up to 2 GB of space. With that in mind, it’s important to watch what you put up there, so you don’t run out of space.

Text files don’t use much space. Graphics, videos and 3D models can use lots of space.

One strategy is to convert your .jpg graphics files to .png format using something like Gimp. Fortunately, the screenshot program under the XFCE desktop, that I use, outputs .png files by default.

The same goes for video. Convert big video files to a compressed format before putting it into your git system.

An alternative is to simply stuff your big graphics, video and 3D model files up on a web server somewhere and then put their web links into a text-based list (or HTML) and keep that in the git environment. You’ll have to manually update the files, although that may be OK since these files might not change very much over the life of the project.

Wrap up

In this article, I showed that you that git can handle different types of files, other than just text. You’ll have to decide which files you want to track with git.

Also remember that you are limited on your project space, depending on the account type, upon a git service site like GitHub or GitLab.

Take a look at the following links, to see the layout and files types others put on their GitHub pages.

Contact Rob “drtorq” Reilly for consultation, speaking engagements and commissioned projects at or 407-718-3274.

A homebuilt Dr. Torq nameplate.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: Torq.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.