I recently dug into the new proof-of-work algorithm powering mining for the Monero crypto-currency, RandomX. It was rolled out to the Monero blockchain early in December of 2019, replacing the CryptoNight algorithm that had been used before it. Several other crypto-currencies are prepping to follow Monero down the RandomX path, so I thought it might be worth investigating. For more background on proof-of-work algorithms, see here.
One of the problems with proof-of-work algorithms and crypto-currencies is that the money reward for faster processing creates a positive feedback loop for the first ones to optimize. The first optimization turns into an increase in income which then makes it easier to make the next optimization, repeat. This usually results in most of the mining rewards going to a few large miners while being prohibitively expensive for a newcomer to get started.
Several crypto-currencies have made it a point to ensure that consumer hardware is competitive when mining to keep the mining workload distributed, which theoretically makes the crypto-currency more secure and definitely makes it more approachable. Ethereum and Monero have been particularly good at this with Ethash and CryptoNight, respectively, working well via consumer GPU computing and resisting FPGA or ASIC optimizations.
RandomX is the latest attempt which initially appears to be resistant even to GPU optimization. Is does this by making maximum use of the typical CPU architecture so that effective hardware optimization virtually requires designing a full processor. This is a costly process which standard CPU manufacturing offsets with volume sales.
Performance
Here’s a quick look at the performance of this algorithm on three different systems:
I built the xmr-stak-rx 1.0.4 miner from source that can be found here. Make sure to apply the large memory page optimizations. You’ll need at least 4GB RAM and 2MB L3 cache.
System
Performance (kH/s)
System Power (w)
Ryzen 2700
4.8
105
Ryzen TR 3970X
24
355
EPYC 7502P
28
235
Structure of a Hash
The phases of the hash proceed as:
Initialization: AES-based scratchpad (2MB) fill
Program generation – AES-based hash (typically hardware accelerated)
Program compilation (for JIT-enabled hasher, much slower without)
Program execution
Results mixing
Here’s a breakdown of where the per-hash time is spent on the Ryzen 2700:
Phase
Percent of Execution
Program Generation
0.24
Program Compilation
1.96
Program Execution
93.84
Results Mixing
3.95
As we can see, the majority of time is spent in program execution. If we’re going to make significant improvements, it’s likely going to be there.
The number of things happening looks roughly like this:
8 programs generated and executed
2048 iterations of
256 randomly generated instructions including an average of 25 loop instructions
Average of 620 iterations per loop instruction (measured)
Each instruction run in an inner loop instruction runs an average of 1.2 million times. Every other generated instruction runs 2048 times. If we find a clever way to move these instructions out of their loops or remove them entirely, there may be noticeable savings.
In the next couple entries, I’ll discuss the results of attempts at improving hash execution time.
I’ve spent some time with ReactJS recently (and in the past), and I thought I would make some notes.
ReactJS is a UI management framework created for managing the presentation and interaction of UI components on a web app (or native mobile applications with ReactNative). It manages the structural document changes behind the scenes and renders them to the browser/screen.
There is a paradigm called Model-View-Controller (MVC) for UI development. The Model represents the data layer, the View represents its display, and the Controller represents the code used to mediate the two. I’ve said in the past that if the Model were rich enough, the Controller would be unnecessary.
ReactJS fills at least the View side of the MVC paradigm — possibly even the View-Controller. There are some standard data-management libraries that frequently accompany it, but they are optional, and I have not used them.
For each ReactJS UI component, you follow the lifecycle through the construction, presentation, maintenance, and teardown of the bits that get rendered to an HTML document.
Despite my love for the functional model, UI design is fundamentally an exercise in state management. There is no user interaction with anything that isn’t changing small stateful pieces one at a time to achieve some effect. However, I would say React does a good job of delineating where state change should happen vs. the side-effect-free computations.
The maintenance phase of each component is very unstructured. State updates happen either through internal updates (this.state) or through the owning component (this.props), and it’s up to the developer to wire the downstream effects together via JS code. The one exception to this is the update of the presentation, which always happens if the component state changes (except in rare circumstances).
In the past, I built small UIs of communicating state machines. React would have been a great tool to have to help with the management of adding and removing components, but that’s about where the benefit would end. I was ultimately going for much more granular control of the UI component interactions. I would rather spell out explicit data flows and state changes in a model than have them implicitly buried in blocks of code.
I think React has the potential to be the foundation of a much richer UI management framework. There are frameworks like AngularJS and VueJS that I’m much less familiar with that may already do what I’m looking for. I’ll have to check them out at some point. My preference for the “minimal” option took me down the ReactJS path, and I like it.
It’s a good exercise to ask, “What’s the smallest thing that would work?” Understanding the limits is a key step in the creation of anything.
In that spirit, what is the minimum description of software? This question has been the domain of computer language designers for decades. What’s the minimum syntax? I think Lisp can be declared the winner. What’s the minimum required description of a language? That’s still being worked out.
What is minimum required computing interface? That one we might be able to answer. The public clouds are touching on the answer in the form of AWS Lambda or Azure Service Fabric — inspired (I assume) by the general trend toward micro-services. There is a notion of an event that can be accepted, and a pre-defined computation that should be executed with the event. In the case where no event is required, you just have a computation that needs to be run to completion.
Any computational job could be described as an event to respond to and a job to run. Event-driven programming is a powerful paradigm that has been around for a while. It requires an effective way to express kinds of events and some kind of program to accept them.
This is a very light definition, so it’s an attractive interface. However, there are some aspects of computing that it doesn’t take into consideration. The biggest one is the run time of the job. For example, it’s often the case that if you can process one event in 1 second, you can process 10 events in 5. This is often the case in database or distributed applications where your computation involves the creation of lots of intermediate sets. There are caching techniques to try to mitigate the problem, but often it’s much more effective to do lots of the same things together. This suggests that the interface needs to be expanded.
n is the number of elements to be sorted k is the key size in bits; the smallest k for n uniquely identified elements is log(n)
I was taught in school (back in the day) that general sorting, usually assumed to be comparison sorting, is O(n × log(n)).
An alternative and well-known form of sorting, radix sort, which places elements relative to the value of the key rather than swapping after comparisons, is O(k × n). Since the smallest-case k for a set of uniquely identified elements is log(n) bits, O(n × log(n)) for radix sorting is still accurate for a smallest-case key size.
However, in comparison sorting analysis the key size is often ignored and the comparison time is assumed constant. Thus the fastest comparison sorts are O(n × log(n)) comparisons. If the same strategy were applied to the analysis of radix sort, its time complexity would be O(n) — k would be a constant factor.
Given that most general-purpose comparison sort algorithms accept arbitrary objects with arbitrary comparators, comparison time is certainly not constant. If arbitrary key size is considered, sort time for comparison sorts would be O(k × n × log(n)) or, for the smallest-case k, O(n × log²(n)).
Radix sort doesn’t readily handle arbitrary data-types — typically used for sorting of fixed-size integers. The argument is that since radix sort isn’t general, its performance doesn’t translate to general sort performance.
What is general sorting? Any data structure that can be represented in computer memory can also be represented as a string, or a loosely-structured, arbitrarily-long sequence of bytes. So string sorting is general sorting.
There is a string sorting algorithm, Burstsort, that efficiently sorts strings and is O(k × n), where k is the average key size. As mentioned above, strings can represent arbitrary data structures, so Burstsort is general for sorting. A general implementation of Burstsort would take a set of objects and a serializing function as input rather than a comparator. The serialization of the objects to be sorted may not be particularly fast, but it is O(k × n).
Therefore, general sorting is O(k × n). Since that’s also the minimum time required to at least look at the keys to be sorted, general sorting is also Θ(k × n).
I’m not sure if this is old news to everyone, but it only just hit me.
I’ve spent the last few weeks playing with OpenStack. Here are some of my notes. I should add that this is almost certainly not the best quick-start guide to OpenStack online. If you know of a better one, please leave it in the comments.
OpenStack is an enormous project composed of numerous core and more ancillary pieces that has been around for a few years. Because the project is so big with so many active players, there have been volumes written about it and narrowing search results to the specific topic you’re interested in takes some effort. There’s also a lot of old documentation that’s no longer relevant that turns up in searches for fixes to problems. Sometimes, a lack of information is more useful. It’s daunting to get started to say the least.
The use-cases I care about:
Single-user on a small cluster.
Small business (multi-user) on a small cluster.
I write a lot of software, most of it libraries and user tools. I want an expandable test environment where I can spin up arbitrary VMs in arbitrary operating systems to verify that those different tools work. As far as I can tell, OpenStack is the only free option for this. I also run some services for which Docker would probably suffice. However, Docker isn’t OS universal. If I have an OpenStack cluster available, I know I have my bases covered.
Note: it’s important to do the above on the demo project with the private network. The public network on the admin project doesn’t have a DHCP service running, so CirrOS won’t connect to the network.
To get SSH working, assign a floating IP to the new instance via:
Network -> Floating IPs, hit “Associate IP to Project”
This is a host administrative task. It allows real network resources to be allocated to an otherwise virtual infrastructure. Then the Project administrator can associate the IP to a particular instance.
Console
Under Compute -> Instances, each entry has a corresponding console on the right.
You can also look up the console URL via (note, must establish admin credentials first):
source devstack/openrc admin admin
openstack console url show <Instance-Name>
DevStack is not intended to be a production installation but rather a means for quickly setting up OpenStack in a dev or test environment: https://wiki.openstack.org/wiki/DevStack
As such DevStack doesn’t concern itself too much with restarting after reboot, which I’ve found to be broken on both Ubuntu 16.04 and 18.04.
The following script seems to bring the cinder service back into working order, but since this is not a use-case verified by the DevStack project, I worry that other aspects of node recovery are broken in more subtle ways.
Bash-formatted script:
# Reference: https://bugs.launchpad.net/devstack/+bug/1595836
sudo losetup -f /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
sudo losetup -f /opt/stack/data/stack-volumes-default-backing-file
# This also seems to be a necessary step.
sudo service devstack@c-vol restart
Note: there’s a lot of old commentary online about using rejoin-stack.sh to recover after reboot, which no longer exists in DevStack — one example of the difficulty of finding good, current info on this project.
Server run-state recovery didn’t work well out of the box. All running servers would come back SHUTOFF when the single-node setup restarted. I looked through many options and documentation, but couldn’t find the right settings that would bring the servers back. Ultimately I created a new systemd service that would record and shutdown all active servers on shutdown, and restart on startup.
Server run-state recovery doesn’t work well out-of-the-box. All running servers would come back SHUTOFF after a single-node restart. Getting them running again has proven to be tricky. There are some mentions of nova config switches that might help, but I’ve not yet seen them work. To make things more complicated, immediately after reboot nova thinks those instances are still running. After some time the nova database state catches up with what’s actually happening at the hypervisor level.
In an attempt to get this working, I’ve created a pair of scripts and a corresponding service to record and shutdown running instances on system shutdown and restart them on system startup (see: https://github.com/joshfuhs/foostack-1/tree/master/code/systemd). However, this doesn’t always bring the instances completely back up, and when it doesn’t those instances seem to be unrecoverable until another reboot. Regardless, this feels like a complete hack that should not be needed given what the OpenStack system does.
I’m giving up on server run-state recovery for now. The success rate is better than the fail rate, but the failure rate is too high for any real use. In a multi-node environment, I suspect the need to completely shut down VMs will be rare since they could be migrated to another live node. I’ll revisit this when I get to multi-node.
Yet another problem with rebooting a DevStack node is that the network bridge (br-ex) for the default public OpenStack network doesn’t get restored, cutting off access to instances that should be externally accessible.
These are lengthy, and I’ve yet to find anything in-between DevStack and this. This might be a weakness in OpenStack. It’s built by large IT departments for large IT departments. DevStack is great in that it lets me play with things very quickly. If there’s an opinionated installer that is also production-worthy, I would like to know about it.
https://wiki.openstack.org/wiki/Packstack – Use Redhat/CentOS and Puppet to install OpenStack components. Unfortunately I’m kinda stubborn about using Ubuntu, and the introduction of Puppet doesn’t really sound like a simplifying factor. I could be wrong.
Python API
<old_thoughts>
The best way to describe the Python API is “fragmentary”. Each service — keystone/auth, nova/compute, cinder/storage, neutron/network, glance/storage — has it’s own python library and a slightly different way to authenticate. On Ubuntu 18.04, the standard python-keystoneauth1, python-novaclient, python-cinderclient, python-neutronclient packages do things just differently enough to make interacting with each a whole new experience. The failure messages tend not to be very helpful in diagnosing what’s going wrong.
The online help is very specific to the use case where you can directly reach the subnet of your OpenStack system. In that case, the API endpoints published by default by a DevStack install work fine. In testing, however, I tend to sequester the system behind NAT (predictable address and ports) or worse, SSH tunneling (less predictable ports), making the published API endpoints useless (and in the latter case, not easily reconfigurable).
Each API-based authentication method has a different way of handling this:
cinder – bypass_url
nova – endpoint_override
neutron – endpoint_url
and in some (all?) cases, explicitly specifying the endpoint doesn’t work with standard keystoneauth Sessions
</old_thoughts>
At least, that’s what I thought until I dug through the Session API. At the time of writing, DevStack doesn’t supply Keystone API v2 (it’s deprecated), and python-neutronclient can’t work directly with Keystone API v3. That prompted me to dig quite a bit to make the Session API work. Once I discovered that the ‘endpoint_override’ parameter of the Session API was how to explicitly adjust URLs for communication, it made consistent authentication much easier. I didn’t find any tutorial or help that would explain this. It just took some digging through code.
There are still some API-style differences between the libraries, most notably neutronclient, but the Session API cleans up most of the initial complications.
I’ll post API-exercising code in the near future.
Current Status
I knew going into this that OpenStack was a complicated project with a lot of moving parts. I had expected this effort to take a couple months of free time to figure out how to (1) setup a single-node system, (2) develop some automation scripts, and (3) setup a multi-node system. Three months in and I’ve got (1) partially working and a rudimentary version of (2).
After all of this headache, I can definitely see the appeal of AWS, Azure, and Rackspace.
I have a little more stack scripting work that I want to do. Once I’ve seen that all of that can be done satisfactorily, I’ll look into setting up a multi-node cluster. Stay tuned.
I think it’s time for a configurable mobile device platform. Like ATX of the previous generation, this could serve as the foundation upon which many fun projects are born.
As the industry fights for thinnest and lightest, they’ve shattered many form and function boundaries. Form and function are separating. Function no longer requires the volume, mass, energy, or cooling of past technology. As this separation progresses, we have a lot more leeway on form, but we seem strictly focused on sleekness. A standardized mobile form might cost some volume and weight, but could still be well within the state-of-the-art dimensions of a few years earlier.
I write this on an Asus UX305FA, an impossible device a mere decade ago. The thin-and-light notebook market has exploded with models and options, all of which have a slightly different mix of features, none of which seem to match any particular person’s needs. Most probably aren’t terribly successful products.
The die-hard DIY crowd has put together projects involving clusters of Raspberry Pis and other groups of small computing units like Intel’s NUC. People are making this work, awkwardly, without any particular standard. The motivation for creative exploration is there, but the industry isn’t facilitating as well as it used to. This slows creativity.
Project Ara by Motorola and then Google looks like a good swing at this idea, but they may be thinking too small — smart phones might be the wrong target. The one place where space is at a premium is in your pocket. However, tablets, notebooks, and even high-performance systems could benefit from a fresh look at smaller standardized form factors.
Of course, with this idea comes all the traditional problems of customizing systems of “standard” parts. Components could be slightly wrong on the spec, parts could conflict in unspecified ways, you’re responsible for whatever monstrosity you manage to cobble together yourself. However, the opportunity for creative experimentation is enormous. Companies like Dell, HP, and Compaq grew up supporting their particular grouping of standardized components. There is still a healthy market of PC customizers and modders. One need look no further than YouTube channels like LinusTechTips to see the enthusiasm, both on their part and the part of their subscribers.
With a mobile component interface standard, component manufacturers would have more freedom to experiment on their own in their domain. Phone, tablet, notebook, and stationary case manufacturers could experiment with all kinds of forms what wouldn’t necessarily survive the design process of a mass produced device. The same is true of the components themselves.
This certainly wouldn’t make the large manufacturers like Apple, Asus, and Toshiba irrelevant. Someone still needs to push the leading edge of technology. It might even make their products better. Their current innovations seem to be more miss than hit. A customization community might provide an idea pool from which they could refine the next major features.
If there are other groups pushing in this direction, I would love to hear about them.
Other notable authors have already written good posts about this topic, but I was recently encouraged to do so as well.
First, the title isn’t entirely true. I’ll break out gdb when I need to get the backtrace of a native application crash. I’ll do the equivalent for any runtime that doesn’t provide information about the method or function that produced the exception. However, I otherwise avoid them.
Debuggers make it possible to make sense of larger sets of code than you might otherwise be able to. This is helpful, but it can lead you into believing you can deal with more complexity than you can. Debuggers are a crutch that get you past some of your limitations, but when the limitations of the debuggers are reached, you may find yourself in a briar patch.
Complications
Threading and multiple processes
Stepping through multiple threads can be a bear. Threading and multi-processing in general can be dangerous to your health. I prefer concurrency models that isolate to an extreme degree any concurrency primitives. I’ve not tried but I’ve heard good things about Threading Building Blocks.
Runtime Data Models
Investigating data in a debugger may require some familiarity with runtime representations of data structures rather than interacting with your data structures via familiar interfaces.
Additionally, the runtime data structures typically don’t have standard implementations, so they can change from version to version without warning. In fact, depending on the goals of the implementation, the underlying form of the data could change run-to-run or intra-run. I tend to prefer to rely on the documented interfaces.
On the other hand, debuggers provide a good way to get familiar with runtime data structures.
Complex Failure Conditions
When using debuggers in my more youthful days, I found that complicated issues were easier to track down if some code changes were made to provide breakpoints under odd conditions. This seemed antithetical to the purpose of a debugger. Maybe I was doing it wrong…
Not Recorded
I’ve never seen anyone record a debugging session such that they could return to a particular debugger state. I’m sure it could be done, but I don’t know how valid that technique would be if small modifications were made to the original software and retried.
Debugger Unavailability
In some rare circumstances, debuggers aren’t available for the environment you’re using. If you haven’t developed any other techniques for finding problems, you may be stuck.
Preferred Techniques
Unit Testing
I cannot stress this enough: unit testing, done well, forces you to break your code into smaller, functional units. Small functional units are key to solid design. Think about the standard libraries in the language(s) you use: the standalone functionality, the complete independence of the operation of your code, the ease with which one could verify their correct functioning. That’s how you should be designing.
Note that once you know the method/function in which the fault happened, you’ve narrowed the problem down significantly. If you did a good job at functional decomposition, you’re typically only a few steps from the source of the problem. If not, judicious application of the other techniques will tease it out.
If you find that the behavior of your module is too complicated to easily write unit tests, that may be a sign that your module is too big. On rare occassion the input domain is so rich with varying behaviors that directly testing the interesting input combinations is impractical. Those can be managed with more advanced testing techniques, see: QuickCheck.
Invariant Checks
When code separation proves to be difficult because of a lack of regression testing around the parts that you’re changing, invariant checks are a technique that can be used as a temporary shim. These are tests that run inline with the code of interest to check conditions usually before and after a method/function call or a block of code.
The invariant code can form some of the foundational functions when you do eventually get around to creating the unit tests for your new module.
Print/Log Debugging
The dreaded printf() debugging! This isn’t ideal, but it can give some quick info about a problem without much fuss. If you find you’re adding hundreds of printf()’s to track something down, I might suggest that you’re using some of the other techniques inadequately. If this is the position you’re in, a debugger might actually be a benefit.
Note that, again, all code written to try to weed out your problem may be valuable for one or more tests.
I’ve also used ring buffers embedded in devices that store the last 1000 or so interesting events so that when an obscure failure happens, there exists some record of what the software was doing at the time.
Where Debuggers Shine
Compiler debugging. This is an absolute pain without debuggers, and I wouldn’t recommend it.
Heisenbugs: Those bugs that disappear with any change to the code. Then a debugger is about the only way to attempt to get a look at it. Those bugs usually warrant pulling out all the stops. Fortunately, many of the newer languages have entirely eliminated these bugs. Good riddance.
Other Tools
I do appreciate other tools like code checkers and profilers. They usually work without much input and communicate their results in terms of the language they’re checking. I’m a fan of this model.
A seemingly close relative of debuggers, the REPL tools, look promising. I’ve never used them, but they look like they operate almost entirely in terms of the language that they’re debugging. I’m a fan of this model.
Summary
I prefer debugging techniques that produce additional useful work and provide support for refactoring if the situation warrants it. Every bug potentially reveals a need to re-implement or re-design. Debuggers feel more like patchwork.
Engineering disciplines have been using revision control techniques to manage changes to their documents for decades if not centuries.
With computers we get new, automated, and more comprehensive techniques for doing this.
Advantages
You have a history of changes. Where did that file go? Look through the history. What changed between this revision and the last? Some revision control tools will show you exactly what changed.
They can coincide with and support your backups, some completely passive. Depending on the frequency with which you do backups, you may already effectively have a form of revision control, though it might be difficult to get some of the related features.
For engineers and other professions, it’s sometimes incredibly important to know what you were seeing at a particular point in time. Some revision control tools give you exactly that.
Disadvantages
Space. Recording revisions usually requires extra disk space. Disk space is relatively cheap, and most revision control systems are far more effective than recording multiple full copies of the same file(s).
Complexity. Controlling revisions isn’t as simple as just writing contiguous bytes to disk. Unfortunately, filesystems aren’t generally as simple as just writing contiguous bytes to disk either. Computing capabilities are solid enough that this is usually a minor concern.
Classification
There are probably two broad categories into which revision control tools can be divided, passive and active. Each has its own advantages and disadvantages.
Passive
As the heading suggests, these revision control systems require minimal interaction to make them do their job.
Some passive revision control systems include:
Dropbox – Look at the web UI. You can easily find older revisions of files that you’ve stored.
ownCloud – This is an open source, self-hosted web tool similar to DropBox. It’s supported on all major operating systems and has apps for every major mobile OS. I use this at home.
Apple Time Machine – Apple provides a way to periodically backup to a secondary drive and provides the revision history of those files. There are similar tools for Windows.
Copy-on-Write Filesystems (CoW) – Several filesystems offer revisioning as a core capability based on their underlying data model.
Most of these will not record every change but will instead catch changes that fall below some roughly-defined frequency. Revision recording throughput is affected by the number of files that have changed and the total amount of extra data that would need to be stored. Because these snapshots are taken without user intervention, there’s really no chance to augment the changes with additional information or to group related changes in meaningful ways.
However, with just a little setup — pointing at a server or drive, entering account credentials, choosing a directory to sync — you can rest assured that changes made to your files will be recorded and available in the event of emergency or curiosity.
I believe every computer shipped should come with some form of passive revision controlling backup system out-of-the-box.
Active
Active revision control systems offer much more capability at the expense of learning curve. However, there’s simply no better way of working on digital (is there any other?) projects.
Some active revision control systems include:
Subversion – an old favorite, but slowing deferring to
git – distributed tool that’s consuming the world
cvs – “ancient” predecessor to subversion
This list is far from exhaustive. I can think of at least five or six others off the top of my head, but I don’t think any others have nearly the significance today.
The features that the various active revision control tools offer are vastly varied, but they all provide a core set of functionality that distinguish them from the passive revision control tools.
Explicit commits. Every bit of work committed to the revision control system is done so explicitly. Specific changes can usually be grouped. Comments and other meta-data can be included with the commits to provide extra context for the change.
Change diffs. Every modification made can be compared to the previous version and changes between versions can be viewed.
View history. Every commit and meta-data ever made can be listed.
Checkout previous revisions. It can often be helpful to look back in time to find out why a problem didn’t seem to exist in the past or to determine when it was introduced. In rarer circumstances, you might want to know why a problem seemed to disappear.
Revert commits or to a previous revision. Sometimes changes were committed that were ultimately detrimental and should be removed.
Multi-user commits. Virtually all active revision control systems support accepting work from multiple users with techniques for merging changes that can’t be trivially combined.
Like the passive revision control systems, not all active revision control systems are also backups. In most cases you would need to take extra steps to backup the revision control system. Pairing an active revision control system with a passive revision control system could be a way to do this.
Few, if any, of the active revision control systems handle binary data well. They can usually be handled, but the efficiency of storage might be lacking and the diff capability is usually absent. This might be their single largest weakness.
No significant (or insignificant) project should be started without one of these revision control tools, and project tools should be structured in a way that allows independent, verifiable revision control.
Visualization
Most of these tools don’t seem to provide much at-a-glance functionality, and I think it’s really useful to have. The things I’m most interested in seeing:
Have any files been modified (active)?
Are there files that could be lost (active)?
Are there any upstream changes that aren’t synced (active, especially useful for multi-user projects)?
Are there any local files that haven’t been recorded (passive)?
For the active revision control questions, tools like Github Desktop for Mac and Windows, TortoiseGit and TortoiseSVN for Windows, and RabbitVCS integration for Nautilus on Linux might do the trick. Some active revision control systems provide these features out-of-the-box, but they tend to be pricey.
On (4), I’ve not seen a passive system that provides this information. It seems like it might be useful to know if all local files have synced before shutting down for a while. I’ll keep my eye out for this.
For those with a bash habit, I have a version of ls that provides (1) and (2) above. I plan to make this available shortly.
As the world moves toward AWS, Azure, and Google Cloud, I would like to take a moment to reflect on areas where the cloud services maybe aren’t so strong.
Connectivity/Availability
While the popular cloud services are the best in the world at availability and reliability, if your application requires those qualities at a particular location, your service will only be as reliable as your Internet connection. For businesses, high-quality, redundant connections are readily available. For IoT-in-the-home, you’re relying on the typical home broadband connection and hardware. I won’t be connecting my door locks to that.
When disaster strikes, every layer of your application is a possible crippling fault line. This is especially the case when your application relies on services spread across the Internet. We saw an example of this recently when AWS experienced a hiccup that was felt around the world. The more tightly we integrate with services the world over, the more we rely on everything in the world maintaining stability.
Latency and Throughput Performance
If latency is important, locality is key. Physics can be difficult that way. Many attempts at cloud gaming have fallen apart because of this simple fact. Other applications can be similarly sensitive.
Likewise with bandwidth, even the best commercial connections are unlikely to achieve 10Gbps to the Internet. If your requirements stress the limitations of modern technology, the typical cloud services may not work for you.
Security
While the cloud providers themselves are at the cutting edge in security practices, the innumerable cloud services and IoT vendors built atop them often aren’t. Because of the ubiquity that the cloud model presents, it’s getting more difficult to find vendors that are willing to provide on-premise alternatives — alternatives that could be more secure by virtue of never shuffling your data through the Internet.
There’s also the simple fact that by using cloud services, you’re involving another party in the storage and manipulation of your data. The DoD types will certainly give this due consideration, but events like CelebGate (1 and 2) are indications that maybe more of us should. Every new online account we create is another door into our lives, and the user/password protected doors aren’t the most solid.
Another concern along these lines is legal access to your data. If you’ve shared data with a cloud provider, can it be legally disclosed without your knowledge via warrant? Can foreign governments request access to any data stored within their borders? These issues are still being worked out with somewhat varying results around the globe. This might be an even smaller concern to the typical user, especially for those of us in the US. However, I feel I would be remiss if I didn’t note that politics of late have gotten…interesting.
Precision Metrics
I haven’t heard about this problem recently, but it has been noted that it’s difficult to get good concrete metrics out of at least some of the large scale cloud providers. Software failures can occur for reasons that are invisible to the virtualization layer: poor node performance due to throttling of nodes in particular locations, odd behaviors manifesting between two VMs using resources in unexpected ways, etc. Good luck getting the hardware metrics needed to debug these from the cloud providers.
Cost
This is fast becoming a non-issue as prices race toward zero. However, at the time of writing, those AWS bills could still add up to a significant chunk of change. Delivering the kind of reliability that the cloud titans provide isn’t easy, and the cost can often be worth it.
Custom Hardware
This is a fairly extreme case, but customization with the big services is rough unless you’ve got significant cash to throw at the problem.
Scale
If you’re already the size of Amazon or larger (Facebook), you’re probably better off doing something that works for you. I don’t imagine many will be in this position.
There you have it. The best set of reasons I have for avoiding the cloud.
If none of those reasons apply to you, and high-availability, high-reliability, available-everywhere is what you need, nothing beats the cloud computing providers. They’re ubiquitous, generally easy to use and automate, and are getting cheaper every day. Sometimes you can even get better deals with specialized services like Bluehost or SquareSpace, but beware the drawbacks.
However, if you have a concern along any of the lines above, it’s at least worthwhile to take a second look.
In a couple of previous posts I’ve provided what I believe to be a good definition of a functional language (Functional Programming Definition) and why a functional model is ultimately required due to the constraints of reality (Everything is Functional).
However, I don’t know that I’ve been clear about exactly why functional programming is so beneficial. I think the benefits can be summed up in one word: predictability.
When I’m working with functional code, I know that I can lift any part from any where and get identical behavior given the same inputs wherever I may need it. When I know the language enforces these rules, I can be supremely confident in this quality of the code, even unfamiliar code.
When working in languages like C++, Java, and Python, I go to some lengths to try to maintain a functional quality to my code. It can be done with most languages. However, when veering into unknown code, my confidence in my ability to modify without odd unexpected side-effects drops dramatically, and it makes those changes much more of a slog. As bad a practice as it is generally recognized to be, I still bump into the occasional global mutable variable or static class member. It doesn’t take many of those to wreck your day.
I think a second closely-related benefit of the functional style is composability. When you have supreme confidence in your components, putting them together can be a snap. This benefit has a little more dependence on good architecture and design practices, but outright avoidance of the features that make code-reasoning difficult will certainly have an impact.
In reference to my earlier post, everything is functional if you’re willing to zoom out far enough. You could start up another process, container, VM, or cluster to restore your faith in the predictability of your code. Note that this approach has costs that could be fairly easily avoided by starting functionally from the start.
Disclaimer: it’s always possible to write bad code in any language or style. However, it can be argued that it’s much more difficult to reach the spaghetti-horizon where the functional style is enforced.