Transcription

Okay, so I'm Ryan Tomego, and this is the Shell Haters Handbook, and Josh told me we were going to have some sick AV here today, and so I figured I'd take advantage of it and do the monster, what, 20-foot ASCII art, so I don't know, I think it turned out pretty good. I'm here today to talk about the Unix shell, which may seem kind of strange at a Ruby conference, but there was no shell conference, and so, no. No, when I think about the technologies that I use and the languages that I use and get the most value out of, not maybe the ones that are most interesting to me or the ones that I want to learn, but the ones that I actually use the most to produce, the Unix shell ranks up there kind of strangely very high. In fact, I'd say after Ruby and maybe JavaScript, I probably use shell more than any other language. We're shell haters, remember? It's a boom, actually. And so that's a little bit disturbing for me, but also really interesting, and so I started thinking about that, and I don't, you rarely see people talking about shell programming, but it's a very big part of my programming tool chain, and so I thought it would be interesting to talk about. And I'm not a sysadmin, I guess I'm an application developer. I work on Sinatra and Rack and some other Ruby projects that Josh mentioned, and I work at GitHub where I do product work and frontend things, and so I don't really use shell to, I don't deliver products in shell that often, but I really couldn't imagine delivering them at all without, or as quickly without shell. And I think that's kind of interesting. I think it makes it worth talking about a little bit, because it is very much a misunderstood programming language, I think. And so just to hone in a little bit more on specifically what I want to talk about today, the shell has kind of two modes of operating. One is the interactive command line, which I'm sure you're all familiar with. It's, you know, you put in a command or some stuff, and then a command runs and some stuff comes out. If you're lucky, it's like a cow saying words. If you're not lucky, it's like removing all your files or shutting down the internet or whatever. And so this, I feel like, in at least the Ruby world, is pretty well-known and used. You almost can't do Ruby development without some use of the interactive shell. But that's not what I want to talk about today. The shell is also a programming language. And this is where I think it's used much less. And I actually think it's a really interesting programming language. But it's hard to approach it, and there's some problems with it, some kind of fundamental learning challenges with coming up to speed and shell and really kind of understanding it. And so that's kind of the rationale for this talk. Has anybody heard of this pro- oh, this project? Yeah? Yeah, this, RBM came on the scene like maybe a year ago now. I don't know if you even know if it's been that long. And it's really just lit the Ruby world on fire. I mean, it's pretty much just assumed that you use RBM now. Everybody's doing it. If you haven't, check it out. Please do. And a lot of the reasons that RBM is able to do a lot of the things that it does is because it embraces shell and uses the shell to its fullest. You just can't get RBM's features without shell programming. And a pretty large chunk of RBM is shell. So if you want to work on it, if you want to hack on it, it's good to know these kinds of things. Another project that uses shell a lot, a lot of people don't know this, but Git originally was mostly a series of shell scripts. And then some of the core utilities, the plumbing commands were written in C, but a lot of it was written in shell. And then later those would move to C. So it was kind of this classically designed Unix program where you start off with a single command and then you add more commands. And the shell really kind of defines the way that Git would evolve. And the reason that it has so many commands is largely due to the shell and the way that it kind of forces you to program. So that's another piece of software that's near and dear to my heart as far as GitHub goes. Oh, I messed, this is like my one funny slide and I just messed it up. But what I was going to say is that a lot of people, whenever Git first came out, went and looked at it and heard about this new interesting distributed version control system and everything and went and looked at it and saw that it was a bunch of shell code. And they thought it was a joke. They were like, oh, you can't actually use this to get anything done. That's just silly. And then, of course, haters are going to hate. And we're going to do a little bit of that today too. And I think it's important to do that with shell because it does have some issues. And so I think one of the keys to understanding shell and really getting your head around it is to understand that it's not a general purpose language. And if you're looking at it as a general purpose language like a Ruby or a Perl or a Python or a C or any of these kind of languages, then you evaluate it that way and you're going to get a bad result. It's actually a special purpose language that's designed to assemble commands. And I want to kind of illustrate this through an example. So here we have a very simple hello world shell script with a twist. But basically, it's a hello program that takes a single argument and then echoes back hello whatever name you put in. So if you say hello Josh, it'll say hello Josh. It has a twist in that if you say hello world, then it'll tell you how cliche you are. So it's a very simple bit of shell code. But I think we should just take a moment and just hate on this a little bit, right? Like, syntactically, Ruby we're used to a very syntactically pleasing language, a very good language. And so I look at this and I see some things that I really immediately don't like. One of them is the square brackets for the conditional part. I mean, that just seems strange to me. And then there's the superfluous then. We all know that you shouldn't really need that. There's ways around that and it's just taking up lines. Also, the line or the end of the if is fi. And so there's a lot of these kind of things that are just kind of annoying and kind of heartbreaking. You just don't feel right programming this, right? It kind of feels dirty and gross. But I think if you can set the sort of superficial syntax differences aside, you can see that it's actually not that much different. You're still able to express the kinds of things that you would express in a language like Ruby. You're expressing a basic conditional. And so that's interesting. It's important that programming languages have that. So now we're going to make it a little bit more interesting. Let's say the boss comes in and he's poking around and he sees this hello program. And he's a clever guy. And so the first thing he does is he runs it hello world. And of course it tells him he said cliche. And he's got kind of thin sin and a big ego. And so he's not happy about this. He's steaming mad. And so basically it's like I want that removed from the script, whatever. And so you don't really want to remove it because you like it. And instead you add this conditional. And you would say, well, if the name is world and the current user is not the boss, then do the cliche joke. Otherwise, just let him say hello world all day. I mean, who cares? And so here we see another little bit of syntax we're able to do ands in conditional script. But again, it's like dash a. It's just kind of ugly. Why would you do it like that? You're one character away from and. Or do double ampersand or something sensible, right? And so these things just kind of grate on you. But still I think it's, again, I think a superficial syntax difference. You're able to express the things that you need to express. So now let's go a step further. And now you're at lunch and you're talking and you're like, you know what I hate? I hate people that name their computer Mordor. Like that is the dumbest thing. Like why is that? Like half the people I know, their computers, how many people here, their name is the name of their computer is Mordor. Oh, great. So nobody beautiful. And this, this would do nothing then. Okay. So, but anyway, we want to add another thing that if you're, if your computer name is named Mordor, then just always call that person cliche, no matter what they put into this program. Right. And so here is where I start to get like flaming mad about this language, because now we have an, or that's totally different from the end. Now we have an end. That's a dash a an hour or is like this double bar thing. And not only that, but we have another new set of brackets. And so if you're learning this language, the way that I learned languages, which is you go and you look at a bunch of code and you try to use what you know of existing programming languages to reason about this language is syntax. It's extremely aggravating. So let's keep going with this stuff. Now we have a, we noticed there's a bug in this program. And the bug is that if you just say hello with no name, it doesn't really give you a nice message or tell you that you're using it wrong. It just says hello. And that's not right. We wanted to give an error message. So we do an else. And here again, we see another kind of weird syntax thing with the dash Z. That means zero length. We're testing the string is zero length. If it is, then we want to say a usage message and exit non-zero. We see more brackets. It's mostly consistent with the stuff up there. So not that bad. But then again, another, then like 10% of our code at this point are thens, whatever, but let's keep going. So now let's say the boss comes in again and he's like, you know, I really don't like this whole thing where you can say hello to just anybody. Like, I'd rather it be like, if they don't have an account on the machine, then you shouldn't be able to say hello to them or something. And so you're like, okay, whatever, go ahead and add that. And so you check the Etsy password file to make sure whoever you're saying hello to actually is a person on this box. And if not, you fail. Now, this is where I get just insanely mad at this language because now we have another and, but it's different than the one up there, right? Like up there it's like dash A and down here it's two ampersands. And not only that, but there's just a graph and like, there's no brackets around it. So we've got stuff in brackets, we've got stuff outside of brackets. There just doesn't seem to be any rhyme or reason to like how this language is working. And so this can be extremely frustrating. Now, I think that at this point you really have two options. You can kind of throw your hands up and just say, you know what, screw this language. I don't even care. I'm going to go rewrite this in Ruby or something sensible. Or you can, you know, go try to look in the documentation. And I'll admit that for a very long time, I just, I took the first option. I was just like, I don't, I don't even care. I could rewrite this faster than I could learn how it works. But let's say you do go and look at the documentation. So let's say, man, if you do this in Linux, it's like no manual entry for it, which that's actually not that weird. It's part of the built-in shell language. So you're probably looking at the shell interpreter on Mac. If you do, man, you get something slightly better or maybe worse. I don't know, just use out like all these commands and like stuff about built-ins or whatever. And so you really don't learn anything there, but I guess you learn that it's like built in or something. So then let's say you happen to know that bash has help built in. So if for anything that's part of the primitive or part of the base language, you can say help. And then that language element, it'll give you a short synopsis. And then you'll get this. And it's like, if commands, then commands, else commands, then commands, else commands, five. So it's like, at first it doesn't seem too helpful. You read in the description, if commands, this is executed, it says zero. It doesn't really seem like this is helping you that much. But the thing to notice is that look at how the commands is the same, right? So it's if commands and then commands. So that seems to indicate that they're the same thing. The thing that comes after the if and the thing that's in the body of the if are both commands. So that just seems kind of bizarre. I mean, that looks like language syntax and the brackets and things like that. And so let's just say you do man bracket just out of sheer anger. If you do this on your machine right now, you'll actually get a manual page. So you can do man bracket. And that's because bracket is a program. It's the test program. And you can see here under name, you have test comma bracket. So it's like shell has this alternative syntax or something built in, this special way of saying test. Or does it? If you say which bracket, you'll see that it's actually a program on your file system. There's an executable file under bin or maybe user bin depending on what kind of Unix you're on. If you LS it, you can see that it's just a normal file, executable. If you dip it with test, you can see that they're identical. So it's actually not a special syntax at all. It's that you're running a test program and the whole if part were commands. And so we can run this on the command line and just get a feel for how test works. So the actual evaluation of an expression in shell isn't even built into shell. It's a separate program. And that's because shell really doesn't do anything at all except for assemble programs in different ways and do things based on the exit status of those programs. And then the bottom one, I love that just for fun, run the bracket, see that it does the same thing. So if we were to rewrite the final example and instead of using brackets, use test, I think we can see it may not be aesthetically as nice, although I mean it's horrible no matter what you do. But this way I think it's a lot more evident of what's going on, like how shell actually works beneath the scenes. You have if and then you're running a test program with those arguments. And test is what's processing them and determining whether that expression is true or false. The or is shell syntax. And then you have more tests. And so you can kind of, it makes more sense now. It seems like the language is almost consistent with itself. Question? The closing bracket is bullshit. There is no, I actually had a slide in here where I left the closing bracket off. But you can't do that. It'll actually validate that the closing bracket is there, even though it doesn't do anything at all. I mean like it's completely worthless. It's just to make the language look like a general purpose language, which I really think is a shame because it's not a general purpose language. It's a special purpose language meant for assembling commands in different ways. And so just to highlight, here we have the shell grammar. If and then and the ampersands and the or there, those are all parts of the shell. And then everything else are commands. And the shell doesn't really know what they're doing. It just knows that they're exiting or more or less doing different things. And so this is used throughout shell. You'll see while true in same programming language, you think that or in almost every programming language, true is some kind of primitive value type. It means something. In shell, it's a command. It does one thing. It just exits with zero. False is the same way. It just exits with one. And so you can use these commands as part of language elements. You could run a while loop like this. And you can use any commands there. So you can write your own commands. You could write true in Ruby. It doesn't really matter. The shell doesn't care. It's just executing commands and doing things based on it. Here we see a more often used example of test. Again, that's not a special syntax. It's really running a command. There it is the other way. So again, I just think it's kind of mind blowing that shell looks like a general purpose language, but really it's a special purpose language where the fundamental abstraction is the command. Almost everything that the shell knows how to do is either about executing commands, assembling them into pipelines, assembling them into and and or lists, things like that. And that's really all that it does. It doesn't know anything else. And there's a couple different kinds of commands. I think this is this is kind of insanely powerful considering that shell was developed almost 40 years ago, at least the first shell was developed through the 70s. And you have built-in commands which are implemented in the shell. So those don't actually, things like test and true and false are actually today built inside of the shell. So they don't actually execute those commands on disk for performance reasons of not having to start the other process. But they work exactly the same as external programs. So all the show programs that were written at the time when they did execute this, you could bring those into built-ins and they still work great. Shell also has functions. They work a little bit differently than functions as we know them then, though they're basically little mini programs. They take a set of arguments and act exactly like commands. In fact, they have the exact same semantics as commands. You can do anything in functions that you can do in a shell script. So you can have if blocks and while blocks and pipelines or whatever you want. And then you can use that function inside of a pipeline or you can use it as the conditional part of an if statement. The semantics are all the same. Later you could take that function, move it into a file, make it executable, remove the function, and all your programs will continue to work. If you ported that shell script then to Ruby, all your programs would continue to work. So there's like an insane amount of decoupling between the shell language and the commands and how they're implemented. I think that's really powerful and interesting. And lastly, aliases also work exactly the same way. So if you have a shell alias, you can assemble that in a pipeline and do all these same things with it. And they can change any way you want. And so the title of the talk is the Shell Haters Handbook. And I really just wanted to mostly hate on shell, but I figured I didn't want to like make the conference overly negative and stuff. So I figured we'd do at least one fun thing with shell. And I think everybody really likes pipelines, right? That's one part of shell that I think most people really appreciate. And I just wanted to go through building pipelines and kind of the process of developing them. And so we have a simple problem here. Let's say we have an ebook. It's Jonathan Swift's A Modest Proposal. And so I'm assigning it to a variable there because the URL was too long. It was like messing up my wrap and stuff. So anytime you see URL, it refers to that. And so usually what I do when I'm working on this stuff is, well, I should say the problem. What we want to do is we want to know a word frequency count of what words are most used in this work. And so because, I mean, I want to know that all the time. So what we would do is we might start by just grabbing the URL with curl. And now we have that text on standard output. And now we just want to slowly move it toward trying to get that unique frequency count. So we would start by adding a pipeline. And maybe we'd use the tr command. I use that. There's a lot of ways that you could do this. You could use sed. You could use pipe that to Ruby. You could pipe it to curl, whatever. I use tr on purpose because it's a standard Unix utility. And it's something that also Ruby has, right? Like Ruby has a tr method on string. And the tr command has roughly the same functions the same way. It's actually something curl borrowed from shell. Or it's the standard Unix utilities that Ruby borrowed from curl. And so here what we're doing is we're saying anything that's not an alphabetical character, replace it with a line in it, or a new line. And pretty much that's it. The C means not. Or it's a complement argument. It means anything not in that first set of arguments. So we do that. And we can see now that our text is now one word per line. And so we can see, though, that we have some empty lines. We don't want those in there. So maybe we pipe that over to grep-v. So we're kind of just taking it one step at a time and just moving the stream closer and closer to our final destination. And what happens here is now it's starting to get a little bit harder to maintain because this command line is starting to get a little bit long. And so there's a really cool trick if you use bash with the standard line endings. And I think zsh works the same way. You can hit Control-X-E. And that will actually open your current command line in your editor. So in my case, it's Vim. You can see by the sidebar on the left, I've tried to make that look like an editor, as much as I could. That's my line numbers. And so now I'm in an editor. Now I have a little bit more control where I can iterate on this program a little bit more. So the first thing I might do is just break it into multiple lines. There's a couple reasons to do this. One is just because it's a little bit more readable. And two, because it's easier to insert things into the pipeline at any point. It's just a matter of adding a line and adding something to the pipeline. So I've also decided to sort the output. So that's probably a good start. We can start to take a look at what it might look like sorted. So if I run this now, I see that I have each word on a separate line with we can see all the A's there and then actual and agree. And so it's going good. If I do Control-X-E again, I'm back in here. Now I might pipe that over to unique, which takes a sorted input and condenses it down to one line per unique line that's in the input. But there's also a C argument that says, show the frequency of that word. So how many of them were there? And so now we're getting really close. We can see that there are nine A's and one actual. So we're almost there. We just need it to be sorted the right way. We can add this sort-rn and boom, we're done. So there's our top lines. And this is extremely useful in all kinds of ways, log files, database dumps. You can really rip your files really pretty quickly and easily at the command line like this. And they evolve into programs of their own. So the next thing I might do is move this into its own file, make it executable, give it a name, and allow it to take an argument. And so that's the way Shell programs, you get from the command line into Shell programs. And these get more and more complex as you go. And so enough of the lubby-dubby pipeline stuff. I want to get back to the hate a little bit. The other thing that's really, really rough when you're starting out at Shell is there's a big documentation problem. Basically, it's mostly garbage. And there's a couple reasons for this. One, we saw earlier the man and help commands don't always work when you'd like them to. So the reference system that's built into your computer probably doesn't work the way that you'd like it to. Another problem is that there's just a lot of shells. And I want to run through these real quick just to give an idea. You have the early Shells. These moved pretty much in a line. So you had Thompson Shell was the original Unix Shell written by Ken Thompson himself. And then that was improved by Massey Shell. Everybody moved to Massey Shell. And then that was improved by Steve Bourne. And everybody moved to Bourne Shell. Now, Bourne Shell isn't batched. That's an important thing to distinguish. We'll get to batch a little bit later. But Bourne Shell was the first Shell where you could really write fairly robust programs. And it actually kind of set the style of Unix programming of the time. Because you didn't really have general purpose languages like we do then. You had C, which was general purpose, but lower level. The types of stuff that you did in C were build special purpose languages to hook into the Shell to do something. Just quickly, if we could go back to here, look at all the special purpose languages in here. You have the Shell, which is one. And then you have TR. I mean, that's really a special purpose language, right? It's a program that processes a language that says a set of characters and replacements to make one. GREP is definitely a special purpose language. And then you have other languages like AWK and SED. And all those are special purpose. And so that was kind of the way that you wrote a lot of programs at the time. And then there was another round of Shells. And this is where people started to innovate quite a bit. So you had CSH that broke with the basic Horn Shell syntax and added some interactive features looking more like C. Horn Shell really took Horn Shell to the next level. And then you had some other Shells that were focused on performance or part of other operating systems, things like that. And then that brings us to the modern Shells, the Shells that are most typically used today. And probably the most popular Shell, I think, by far is Bash. It's a default Shell on Mac. It's a default Shell on most Linux systems, although not Debian. The last Shell is. But ZSH is also popular. How many people here use ZSH? That's a lot. OK, that's a good number. ZSH is really popular in the Ruby community. It's a great Shell. Its interactive capabilities are definitely a lot better than anything else out there. But a lot of what these Shells did is they added a lot of those types of features. So these Shells focused almost entirely on making the command line aspects of the Shell better. So read line and command completion and all these kinds of things were really developed during this era of Shells. And the problem is that in order to do that, they had to add a lot of new stuff to the Shell language. Especially Bash has just an insane number of new features that aren't included in the earlier Shells for things like arrays and string functions and all these things that were previous to Bash punted off to individual special purpose programs to do. Now those are in Bash. And so in my opinion, this creates sort of a problem when you're trying to learn Shell because it takes what is really a fairly simple thing and makes it complex. Here we have the Bash man page, which is roughly 40,000 words. And the Dash man page by comparison, which is a much smaller Shell, much lighter, doesn't have a lot of the interactive features, it's more of a pure Shell implementation, is only 10,000. So Bash is 40,000 times, four times larger than Dash. And so that creates a problem. I think when I was trying to learn Shell, I'd go into the Bash man page, and it's just like a big mess of stuff that I'm not really interested in, and it seems like it's important because it's there, but it's almost annoying. And so, oh, this is from the Bash man page. They're not, they even know this. I mean, it's too big and too slow. And some people, I personally, I don't like Bash. I like it as an interactive Shell, but I hate it for these reasons that I feel like it's really kind of made it a lot harder to learn Shell and approach Shell. And people say it's too big and too slow. There's some like resource issues there. I don't really care about it as a runtime. I use it to run my scripts. I use it to, as my interactive Shell, but I don't use it for documentation at all. What I do use is POSIX Shell, and I just want to quickly get through this. I'm running over, but so POSIX Shell is kind of weird. It's not an actual Shell. It's not, you can't go and download POSIX Shell or install it, but all of the Shells that I listed on the modern Shell page are POSIX compliant, so they all know POSIX. And what POSIX was, is it was, they went to standardized Unix, because there were like 10 of these things. They were all proprietary, and all the different Unixes were going in different directions. So POSIX came along, the IAAA, and the Open Group, and my 10 other groups, and it's a real compliment history, and standardized all these things. And one of the things that they standardized is the Shell. And they did a really good job of it, because what they did is they took like almost a top-down approach, which hadn't been done before. All these things are just kind of evolving. They looked at the Shell, and they looked at the standard utilities, and they said, what is kind of the minimal amount of stuff that we need to have like a working operating system, and that's also the most common and portable between operating systems. And so at the time, it was all about portability. But today, I don't really care about that. I mean, I'd like my Shell programs to run on Mac. I'd like them to run on Linux, but I don't really care about portability the way that they did. What I care about, is that I have good documentation that's minimal, that I can go in and understand, and discover how to work in this programming language. And so that's what's great about Puzzle Shell. The problem with it, and this is the thing that I hate again about Shell programming, is you can't find the thing. It's like impossible. It's like one of these weird standards that like you go here, and there's like a bookstore, or I'm sorry, like you have a shopping cart, and all this stuff, and it's just kind of a mess. But there's a free HTML version, so you go to that, and then you get like one of these forms. And it's just like, what is going on? I just want to see like some documentation. In the meantime, if you go to Google and search for Shell, there's like all these vast tutorials that are talking about their string functions and stuff. And that kind of information is really easy to get at. And so that's basically another time that I just give up. I mean, screw it. And so I'll save you the trouble. That URL right there is to the Puzzle specification. I'm not going to open it. It's nice. It's like, it almost looks like Ruby's documentation. You have the Shell language, the standard Shell utilities. There's only about 40 of them. That's another big problem. You have thousands of commands on your machine. Which ones are the like core ones used for Shell stuff? And the table of contents is a little messed up. So I threw together a page at this URL, shellhaters.ruby.com.puzzix. And I tried to basically, these link to the Puzzix documentation pages. And I tried to just do like almost like a cheat sheet or an intro. So these are Shell building commands, and then all your standard Puzzix Shell commands. And I can't recommend enough using this document. If you're reading a Shell programming, or you're interested in Shell programming, use this documentation, because it'll save you a lot of time. And it's just introducing you to the things that are really important about these programs, instead of being a big mess of stuff that people have added over the past decade to Bash. And so, thank you. That's my talk. I'm Ryan Tomiko. Thank you very much, Ryan. We can do one question. Yeah, there's actually a lot of things. I don't know if there's any, like sysadmins, or... Repeat the question. What's that? Repeat the question. Oh, I'm sorry. He said, how can you hate on Bash without talking about string receiving? And there's actually like a lot of things that I could have hated on, that I had to cut out. I ran a little long. The first time I put this together, the first time I rehearsed it, it ran for I think an hour and 45 minutes or so. So I was just like, I was like, man. So I tried to put it down, and some of it was pretty good. I had some string receiving. I didn't even talk about setE, which is like a major, major thing in shell programming. The first line in your script should always be setE. It's basically like actSane thing. Look that up in the POSIX documentation. But it just kind of shows how many things that I didn't get to, because that's kind of a critically important part of it. But so I'm sorry. There's a lot of things like that that I didn't get to. Thank you very much. That's it. You can talk to Ryan more on the break.