I have been in search of what you’ve put together. My wife and I have tried unsuccessfully various strategies along these lines that were more lofi, but always come back to the spreadsheet. Another vote for open sourcing! I would be willing to contribute in my spare time to something like this. It’s a big gap for many, and all these beautiful iOS apps fall short in every way (but mainly I can’t bring myself to put my bank info on any of them!!).
PSA: if you take TripleByte's online quiz, they don't have the capability of giving you the results of the quiz. At least, that's what I was told when I took the online quiz -- they couldn't share the results with me because the quiz is "adaptive" and scores and questions vary each time the quiz is taken. (I have no idea how they're able to do anything useful with the quiz results if they don't record them and the scores vary so greatly.)
I have no faith in TripleByte or their process at this point, especially since I feel I was somewhat taken advantage of (I did the online quiz expecting to get some feedback -- isn't that the point?). I'm certain they will use the collected data from my quiz to build their product, though.
I couldn't find this on your product page, but does Hazelcast Jet support "global windows"? I've found 90% of stream processing systems I've come across only work within a window, but if I need to perform a computation "for all time," I'm SOL.
We are adding something called a "rolling aggregation", where you receive a record, accumulate it and then emit the current accumulated value. I'm not sure if this matches what you want.
It's interesting to me that, before I learned about databases (circa 2004), I built 3 distinct software products based on flat files (no database). They were all written in PHP (the only backend language I knew at the time). This was before the days of GitHub, so I don't have links to the software anymore, but I may still have the (horrendous) source code. The products were: a forum (!) system called "DatForum"; a chat app called "Chategory"[1] (this one still apparently exists on SourceForge); and a flat-file backed bookmark manager that synchronized with del.icio.us called "LinkDex." I did also build a blog/CMS based on flat files, which I used personally, but it was hardly productized (practically server-side includes). Obviously, using flat file databases is very different from static site generation, but since all of these were written using flat file databases, it would have been easy to optimize their (often poor) performance by statically regenerating the site periodically, instead of reading the flat file database at render time.
Please tell me that the referenced C# dialect used in Midori is available in some form for use, research or general learning? This would really make my week!!
You have Joe Duffy's blog entries about Midori and one entry back when it was still called M#.
Then there are the little pieces that came to C# from it.
Namely async/await, TPL, the MDIL compiler on Windows 8.x, the .NET Native compiler for UWP, the improved GC control in .NET 4.6 and the planned features for C# 7.1, 7.2 and 8.0 related to more mechanical sympathy.
Cool! I authored a web based accessibility tool called Graybit some years ago. It allowed you to enter a web address and browse the web through its proxy, which converted all CSS, html and images to their grayscale equivalents. The idea was to simulate complete color blindness to raise awareness and encourage higher contrast and things like using underlines for hyperlinks.
Something else people should know: AWS ES is on the Internet. You can't deploy it to a vpc yet, and you can only lock it down using IAM, which may or may not be good enough for your use-case.
For those that prefer the ease provided by AWS ES Service, consider Elastic Cloud, which affords most of the same capabilities but is run by Elastic themselves (it was previously known as Found, which Elastic purchased a few years ago). There's also an Enterprise offering. If you're looking for a hosted Elasticsearch solution, it's probably better than what AWS is offering. Side note: they update about as often as elastic releases, whereas AWS ES is consistently behind.
As Daniel Parker (acloud.guru) mentioned in the article comments, they went with Algolia [1] despite being AWS experts. He cited the uncertainty and complexity around AWS ES as the problem.
An attacker is only ever RCE on one server away from being on your VPC subnet. You're going to have to set up authentication for internal applications anyway, although I suppose vulnerabilities in the login process are harder to exploit if you can't even get to it.
I'm curious about how the pricing compares. I'm not very satisfied with AWS ES, but managing ES manually doesn't seem like the most fun either. (In fairness it looks like there's not too many knobs to turn, but it's still another concern to have to personally deal with.)
The IAM authentication is really annoying. It's not supported by many client libraries, nor have I found an easy way to make arbitrary HTTP calls with signature v4.
The only other options are completely public or IP-based whitelist, the latter which is untenable in most cloud environments.
A simple solution in this vein is to white list your the EIP addresses of your NAT. This would give access to all resources in a private subnet (this is useful for Lambda's running in subnets).
While AWS ES can be cheaper in some configurations, Elastic Cloud is actually quite competitive in pricing for larger clusters when compared to AWS' ES-service. This post compares the two services, and there's an example price comparison at the end of the post: https://www.elastic.co/blog/hosted-elasticsearch-services-ro...
We support most official plugins, and if you get a gold or platinum subscription you can upload your own plugins. Elastic's X-Pack is included in every cluster, which includes security features like role based access control.
It's not possible for external service providers to integrate with IAM at this point.
One issue I've found with Elastic Cloud is that there doesn't seem to be a horizontal scale-out option other than multi-DC or getting bigger boxes. Is horizontal scaling in the works? Easy horizontal scaling seems like one of the better benefits of ES.
Or, alternatively, am I mistaken about how configuration works?
Per availability zone, Elastic Cloud currently scales vertically (in power-of-two increments) until a cluster hits 64GiB memory, at which point multiple 64GiB-nodes are added. While you can run Elasticsearch with e.g. two 8GiB nodes per zone, we prefer a single 16GiB node as there's fewer things that can go wrong. (If you want the second 8GiB node for redundancy, then that's exactly what our multi-zone HA configurations are for, and we encourage HA setups by making them less than twice (or thrice) the price, throwing in additional master-only nodes for free)
(A bit of history: When Found (the company Elastic acquired and which is now Elastic Cloud) was in private beta in early 2012, we actually did allow custom cluster topologies. We ultimately disabled that as it was overwhelmingly used to make sub-optimal cluster configurations, such as 5 x 1GiB memory nodes)
Any idea if/when Elastic.co will support multiple user accounts and 2FA in the management portal? Not having those was a deal-breaker for us when we evaluated it awhile back, and was the sole reason we went with the less stable AWS service.
My understanding is that this is going to change and soon. Amazon has been poaching key elastic search employees presumably with the idea to improve the service to on par or better.
Elastic employee here, and this is the first I'm hearing of that. I don't know a single person that has left for Amazon, from any team. Certainly no "key elastic search employees".
Your question seems to indicate that you have a runtime dependency on github. If true, this is a problem no matter where your git repo resides. You need to architect a solution wherein you do not have a runtime dependency, especially if it is a single point of failure, or accept that your reliability will never be better than the sum of your services' reli abilities (there was an article on the "calculus of availability" or something like that recently).
What I personally do in AWS is bake my artifacts or other git-sourced data into AMI's. If you want a middle ground, you can instead push your artifacts to an s3 target -- s3 has better reliability / track record than github for this purpose.
Thing is, runtime dependencies on Github is something that Github is promoting and selling now (https://github.com/marketplace/). Maybe this incident puts that value proposition in a more visceral context.
How is the GitHub marketplace a runtime dependency? Code review, project management, build bots or whatever else should NOT affect the availability of your website.
see now you're just splitting hairs and getting into philosophy. if your website will be stale tomorrow because you can't update it today, then that is a runtime dependency imo.
Yeah, I should have clarified I was talking with my operations hat on. From a development perspective, you might call it a "deploy-time" or "scale up" dependency.
naive tip: give your app that is relying on github a SKIPGIT environment variable to ignore the bits around git operations, and just make local changes as an interim hack.
>
Your question seems to indicate that you have a runtime dependency on github.
Really he is speaking to the problem with SAAS. Sure it's cheaper to rely on someone else to do the heavy lifting for you and they can do this because they 'make it up in volume'. But the other side of that coin is no one really knows how to do that anymore. If you wanted to roll your own it's going to be very hard for your company to do the basics because you've become dependent upon the cloud, and someone else paying employees to do the basics for you... The basics have never been hard.
But a positiv counter argument would be that github/gitlab also does the sharing part of the jobs and you (hopefully) pay them to handle all the security aspect of that public sharing.
Opening a port to your local instance of git implies more thinking and security consideration. Of course you could host a mirror on a cloud instance, but then it's saas anyway and you have even more work to do before ever starting coding.
So yeah github/gitlab does the heavy lifting for you, but for small organisations that might be cheaper overall than to pay somebody mastering all required stack to implement and maintain your own instance(s). For big company no question that an in house git team is probably more efficient.
Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub? I assume most apps will not go down if GitHub goes down, but their ability to move code to production gets stalled.
Git supports multiple remote repositories. Have github be your main repo, keep a mirror locally or on another service (updated using hooks), and switch to it if github fails.
Speaking from an operations perspective, yes, it's a runtime dependency. From a development perspective, you might think of it as a "deploy-time" dependency. Think of it in contrast to compile errors. If your program has errors that your compiler can catch, then it will refuse to compile the program and you must fix the error before executing the program. A runtime dependency is one that may change after your program is compiled -- these are harder to test, especially when the program may fail depending on its environment. In the case of dependencies on services such as Github at deploy time, it's a "runtime dependency" for the operations programs. Since the operations programs are the ones with the runtime dependency, it's also in the operations team's purview (and responsibility) to revise their program(s) and eliminate the runtime dependency such that testing (and reproducibility of) deployment is deterministic and reliable.
There's the convenience reason. Teams making that trade-off (not having a local mirror) need to understand they made GitHub itself a runtime dependency.
But in theory in a pinch you could spin up a git server somewhere publicly accessible and switch your site to pull from there, and push to there yourself.
Agreed, that shouldn't be too hard if you're really in a emergency and can't wait. What I see though is some deep integration with their API with automatic tagging, releases, etc... That's hard to work around but still not impossible if you're willing to live with some chaos in your process until things are back.
Not if you're using more than git from github, for example downloading releases as zip files for deployment.
There are probably a lot of other tempting features to put on your deployment critical path but I'm not in an environment that uses github so I've forgotten.
Webhooks are a big one: rather than polling a git server for changes it is very easy for CI/CD systems to rely on GitHub's HTTP webhooks in their critical path to kick things off on push/PR/branch/merge. Especially because GitHub does a great job of populating its webhooks with tons of useful information about the event that's tough to replicate with just git post-hooks.
"Becoming"? Can you point to any large project (at any point in the history of Git) that doesn't use some centralized service as a single point of distribution/source of truth?
The beauty of Git's design, though, is that you've still got everything to use it distributed even if you, under normal circumstances, just use it centralized like a fancy SVN.
> Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub?
It's not a runtime dependency, it's a stupidity to rely on random
repositories from the internets when building and deploying (remember the
left-pad farca that halted work for half of the web programmers). You should
always have your local copy of necessary sources or use repositories that can
trivially be swapped with some other mirror.
We used to have the infrastructure for those things (in the form of binary and
source package repositories), but it was mainly used by sysadmins, and
programmers usually have an alergy for such infrastructure.
> you can instead push your artifacts to an s3 target
This is our tactic. I tried to do baked AMIs at one point, but the 10-15 minute turnaround in registering them meant that we couldn't use them for staging or testing (too long to iterate changes).
Previously we were capistrano-deploying with git from bitbucket - every server had to individually git pull from servers on the other side of the continent, over the public internet. Susceptible to all sorts of problems.
You can speed up AMI baking by using chroot + a mounted EBS volume on a running instance. It should drastically reduce the time it takes to produce a new image. Also be sure that you're building EBS-backed AMI's, as S3-backed AMI's will require a lot of time for copy.
I can appreciate how holistic the GNU ecosystem seems to be. I'm a lover and user of Nix, so whenever something about Guix comes up, I draw the contrast. In this case, GuixSD is using Shepherd, which uses Guile for configuration and feels like natural continuity within the system. NixOS, on the other hand, provides systemd (and only systemd). In the case of nix, it's smart to leverage existing technologies, for velocity, yes, but also as a sort of proof that the core principles behind nix can apply even over systems and tools that are completely unaware of nix concepts; on the other hand, GNU really makes me a believer in the fact that Guix is very pure.
Basically, I thought I'd never work again for someone else. That was over six years ago. Mapbox has grown from about 15 to over 220 people now. Somehow I knew it was different and it was just compelling enough — plus I had a "try before you buy" experience through contracting that was just too good to pass up.
On the balance, I definitely enjoyed the software (and still do), but especially working at home on my own, I enjoyed the interaction with people (usually by email or phone, sometimes in person) and the constant variety and problem-solving nature of it. And not just technical problems, but organizational and strategic problems, too.
I think it could have been as lucrative, but at least in the Mac software space (and probably the iOS one more so), it had a certain element of "grind" to it, with product marketing and website presentation, working PR contacts and releases, and constantly getting things in front of influential people. As much as I enjoyed that some of the time, I didn't want to do as much as it would have required to make apps the lion's share of my income.
Thanks! I never thought of it this way before, but I suppose there's no "set it and forget it" to selling on the app store if selling on the app store is your primary source of income, since you have to keep working to ensure the product remains relevant, gains new customers, etc. (Selling on the app store is decidedly not SaaS, after all.)
Yep, exactly. Even the "greats" just relentlessly promote and keep their apps relevant. Even a one-day or one-week windfall from a viral launch will not sustain you over time.
