Skip to main content

Evolution of thoughts on software

November 2012

There are many changes underway that affect software development.

  • Multi-core processors
  • Cloud computing
  • Predominance of mobile devices
  • Smarter embedded devices in everything
  • Ubiquitous Internet

The amount of software is growing, perhaps exponentially. The environment for software is also growing in complexity. At first, all software was in the laboratory and never left. Then with mainframes, the software stayed in controlled rooms. With desktops, the software was on portable media, such as floppy disks. With the Internet, the software moved to desktops and servers as it had in portable media, only over the wire without the media. Now the software is on mobile devices, embedded devices and the cloud. Software development lifecycle is now officially insane.

The software language has not kept up. We are still using Java and C. These languages were designed for a different era, a different set of challenges, just as COBOL was before them. Two new languages look appealing: Lua and Scala. They have new features suitable for a wireless internet world. I especially like the Actor model in Scala for distributed programming.

Efforts such as CORBA and EJB attempted to make distributed programming look and act like local programming. The idea was that the developer would code as if the objects were local, and the brokers would handle the complexity. It does not work it practice. It is not even a good idea. One cannot treat remote programming as local programming, but one can treat local programming as remote. If one assumes that everything is remote, then it is fine if something happens to be local. However, if one assumes that everything is local, then it breaks down if something happens to be remote.

We need a new way of thinking about software. Object-oriented programming was a breakthrough in how we thought about code. It is no longer enough. I believe that new thought model is spime.

In 1995, I was working for the US Department of Defense High Performance Computing Modernization Program, an office of the Director of Defense Research and Engineering. I had the challenge of tracking all of the research usage of the various defense supercomputers and matching it to Congressional funding. Back then, Congress still thought $1.6 billion was a lot of money and they wanted to know how it was used.

The hard part was not tracking the computers or even the research. The challenge was tracking the users. I was struck by an observation: every computer on the Internet has a unique identifier, an IP address, but computers are only half of the Internet. The other half is the people, and there is no unique identifier for people on the Internet. A user is treated by each application on each computer uniquely, almost ancillary like mouse or keyboard.

I proposed that each user receive a unique ID for access to defense supercomputers and that the same user ID be used on all of them. I further proposed that if successful this ID program could be expanded to all defense computers and eventually the whole Internet. Nothing came of the proposal.

In 1996, I was working on a DARPA project called Shaolin, or System of Systems. DARPA was looking at massively connected systems and how to achieve greater battlefield awareness. The DOD wanted to know the status of all the things, such as tanks, trucks and soldiers. I proposed that each thing have a service of the Internet, which I called an avatar for lack of a better word. Cyberpunk novels had discussed avatars for some time. All event messages to and from the thing would pass through the avatar service. The avatar would then be able to establish or recreate the state of the thing.

For instance, if I wanted to communicate a message to a tank system, then I would send a message to my avatar, which would send a message to the tank avatar, which would send a message to the tank system. From my perspective, I sent a message to the tank. The avatars each have the message, so they each know the state of things. If a third party, say my commander, wanted to know if I sent a message to the tank, he would not have to get in touch with me directly. He could send a query to my avatar (through his avatar of course) and get the answer.

I described the concept as two realms: the physical realm and the digital realm. The physical realm I designated with the letter phi (Φ) and the digital realm I designated with the letter Q, for quintessential. In order to achieve battlefield awareness, one needs to create avatars in the Q-realm that correlate to the things in the Φ-realm. The idea of using avatars might conceivably work for blue force things, that is, the things on your side, but would not work for red force things. Nevertheless, it was a start.

In 2003, the company name was Trans-enterprise Integration Corp. RFID was new and exciting. The Department of Defense created universal identifiers (UID) for its inventory, a combination of manufacturer ID, product ID and product serial number. The Auto-ID Labs and MIT ( were actively involved. I also discovered the Dynamic Delegation Discovery System (DDDS), a means for resolving a uniform resource name (URN). The idea was to make web services available by URN. If you had the UID for the object, then you could access services for that object.

In 2005, I was approached by an engineer from IRIS Corp in Ann Arbor to help work on sensor data fusion. He had conceived a fundamental breakthrough but was at a loss on how to implement it. During the project, we worked with the Oak Ridge National Laboratory, the National Geospatial Intelligence Agency (NGA) and the University of Alabama at Huntsville, which is closely associated with NASA, NOAA and the US Army’s missile research.

Sensor data fusion is the ability to resolve that two or more sensors are observing the same thing. Current practice is to have each sensor geospatially resolve its own data before it shares the data. The fundamental breakthrough was to treat location sensors as any other sensor, specify the geospatial interdependency of sensors, and stream the data of the various sensors with a description of the system. The community constantly differentiated remote sensors and in situ sensors. For this approach, all sensors are in situ servers. Geospatial resolution occurs as the last step, not the first. As such, the fidelity of the data and the resolution are dramatically increased. Furthermore, sensor fusion is a central aspect, not an afterthought.

In many ways, sensor fusion is like remote computing. Everyone else was using a CORBA approach. It is possible to assume that all sensor data must be fused and then not have to fuse it. However, it is exceedingly difficult to assume that sensor data will not be fused and then have to fuse it. (There has to be an easier and more eloquent way to express this concept.)

The implementation of sensor fusion had several parts.

  1. The sensor model, a normalized description of how a sensor responds temporally to energy within a particular geometric envelope.
  2. The transducer model, describing the interdependencies of sensors, actuators and processors.
  3. Transducer data transport protocol, with the critical requirement of how to resolve time in nanoseconds.
  4. Transducer discovery, for descriptions, communication and archived data.

I looked to marry sensor data fusion with the concept of the avatar. First, there is a boundary between the Φ-realm and the Q-realm and at that boundary are sensors and actuators. Sensors communicate from the Φ-realm and the Q-realm and actuators communicate from the Q-realm to the Φ-realm. Second, with sensor fusion, there is a means to have both blue and red things in the Q-realm. Sensors create data. Each sensor data element is resolved to a single temporal-geospatial event from the Φ-realm into the Q-realm. These resolved events I call monads. Monads are like grains of sand on a beach. Agents must search and translate these monads into higher level patterns that I call shadows. Eventually, these shadows are associated with an avatar.

For example, a UAV takes a picture. The pixels of that picture turn into monads. An agent scans the monads and detects the image of a tank. That is, a shadow. The location of the tank shadow corresponds to the location of my tank. The shadow is assigned to the avatar of my tank. Situational awareness is established that the UAV saw my tank.

With the explosion of UAVs, the need for sensor data fusion is even more pressing than it was seven years ago. There were many things that arose from the sensor fusion technology that we were not able to pursue.

  1. Very inexpensive synthetic aperture radar (SAR)
  2. Three-dimensional modeling from simple digital cameras
  3. Use of FPGAs for geospatial kinematic calculations
  4. Calibration of sensor models

While we were working on sensor fusion with IRIS, we were also working on service oriented device architecture (SODA) with IBM, Band XI and the US Army. SODA was built on OSGi. The idea was to create embedded software that connected sensors and actuators to the internet as services in a service-oriented architecture. We had a live mission of SODA at Super Bowl XL. Nothing much seemed to come of it. In part, I think, because of the complexity of implementing SODA in Java. IBM created an Eclipse tool that generated most of the code for you, but the complexity of the generated code was overwhelming. SODA divided the work into layers, but forced the developer to deal with all of those layers. The purpose of layers is to avoid looking at them unless you have to.

In 2007, I had Exmark Manufacturing as a client. They are a division of Toro that makes commercial lawn mowers. We devised the idea of a smart mower that would record its usage through sensors and upload that information to the internet. I had recently heard a speaker discussing Bruce Sterling’s concept of spimes, the idea that “one could track the entire existence of an object, from before it was made (its virtual representation), through its manufacture, its ownership history, its physical location, until its eventual obsolescence.” We were looking at making Exmark lawnmowers spimes. Toro has actually implemented the smart mower system, but they have not taken it to the level of spimes.

In 2010, I was dealing with software lifecycle management issues. I documented an end-to-end system. Source code is checked into a source code repository. A build server pulls from the source code repository and artifact repository and builds code based on declared dependencies. Built code is released to the artifact repository. A deployment server pulls artifacts from the artifact repository and deploys functioning software to devices. The concept of organic programming was buzzing around my head. Somehow, I am missing a piece here.

In 2012, Eclipse launched a new projected called M2M for machine to machine ( The primary contributors are IBM and Sierra Wireless. Some of the SODA developers are on the projects. The challenges and opportunities have not gone away. It’s all there waiting for someone to break through.

Here are the ideas so far:

  • Everything is a spime
    • Deployable software components are spimes
    • Devices are spimes
    • Users are spimes
  • Spime is
    • Unique
    • Long living
    • Discovered by name (URN)
    • Semantically logical collection of services
    • Cloud base, not connected to a particular server
  • Clade is a
    • Semantics and ontology of the spimes
    • Schema of the spime services
    • Factory of the spimes
    • Resolver of the spimes
  • Service is
    • Input – process – output
    • Streaming, synchronous, asynchronous
    • What the software provides
    • Provides the state of a spime
  • Spimes exist within spaces.
    • Q-space is the “production” environment.
    • Other spaces would be QA, DEV, UAT, etc.
    • Space is resolved by the DNS

Certain issues that use to be a challenge are now much easier. For instance, before cloud based computing, it was hard to conceive how one would make a spime long living and not connected to a particular server.

Certain issues still gnaw at my thoughts regarding spimes, SODA, sensor fusion and such. For instance, how does one create the software spime? How does one create the user spime? What do they look like code wise? Is it simple enough to be practical? Is it really necessary to have user spimes in order for all other spimes to work? (Something tells me yes.)

Spime, SODA, sensor fusion, it all fits together. Is it too big?