/dev/chat

Jeff Schaller

01:56

semi-idle curiosity, for anyone who has >= 25k fake internet points; do the site analytics here show the kinds of browsers that hit U&L? The help page doesn't say explicitly. A random yahoo article indicate it is/was available. I'm working on a Meta post and I'm wondering what the impact would be.

cas

04:34

@JeffSchaller I've never even bothered to look at the site-analytics before...but it doesn't seem to have any User-Agent stats. It has two tabs. The "History" tab has three graphs, for Posts, Votes, and Traffic. The "Traffic Sources" tab has a pie chart for Traffic Source (divided into referrer, search engine, and direct), another pie chart for Top 10 Referrers, and some lists of Referring Sites and Search Engines.

Michael Homer

The site analytics page has virtually no useful information. It's basically just "which search engines are popular" and some question trends

cas

The number one referer for U&L is duckduckgo.com, with stackoverflow.com a distant second.

user9303970

10:30

Someone wants to review the organization/architecture of my Bash program in GitHub? I invested so much in this, asked so many questions here, I was shouted upon on so many times. I feel this is quite ready (it might have a code problem I'll fix asap) but I think my organization of the file system (and some files) could improve a bit. If you want to give a few words on the code, please do.

This is basically a Ubuntu-Nginx-WordPress environment bootstrapper.

github.com/benqzq/uncwe

Jeff Schaller

10:44

Thank you, cas and Michael!

Kusalananda

@user9303970 The conf/assignments.sh script write to the user's .bashrc. This may be a bit surprising for the user. The local/imb.sh script dumps all databases on the local machine. This may be result in a possible breach of privacy. It is really hard to follow the various scripts as they all use variables defined elsewhere. The scripts themselves also have unintuitive names.

local/tdb.sh kills a tmux session, but it's unclear why (this too may be a bit surprising for a user using tmux).

crontab/crontab.sh replaces the crontab of the user with its own crontab.

crontab/weekly.sh deletes mail?

And log files!

terdon

And all of this needs to be run with root privileges and passes the -y ("say yes to all the things") option to apt, so can cause all sorts of compatibility issues. . .

@user9303970 I would suggest you just keep this for yourself. It's a great way to teach yourself about this sort of thing but is absolutely not something you want to be distributing to others. You know how you want your system set up, so you now have something that does it for you (although I would strongly urge you to remove the -y from the apt commands), so yay!

Kusalananda

install.sh does rm -rf ${repo}/.git but the repository is located in ~/$repo/

Kusalananda

11:04

@user9303970 Do not take the above in the wrong way. I'm fairly used to reviewing all sorts of code and I'm a picky reviewer at the best of times. I do appreciate your questions on the site as they are usually well written. The scripts in the repository, however, presents some issues, mostly for whoever runs them blindly (as root) without knowing what their side effects are.

Suggestions for improvements include collecting some of the small scripts into larger scripts carrying out several tasks (maybe just one script, possibly utilizing local shell functions), and making sure that side effects (such as replacing any crontab for the user and deleting mail and log files) are avoided, or at least are clearly documented.

Also, restricting the operations that needs to be carried out by root somehow would be a good thing.

user9303970

Thanks @terdon I totally agree this suites to be kept to myself. This is indeed meant, by principle, for personal usage. @Kusalananda Much thanks! I wanted all variables to always be available (I sometimes use them for aliases or other purposes and cannot know when I'll need one. the tdm alias is to (temporarily) unlock phpmyadmin hence usage in tmux and killing of tmux session after 2 hours of using phpmyadmin.

I delete all logs and mail each week just not to have accumulative data that will grow and grow (I've seen mail files of thousands of lines and by principle I want to keep the SSD free as possible).

Fixed rm -rf mistake.

Kusalananda

@user9303970 Log files are best rotated by logrotate or similar tool.

user9303970

11:29

@Kusalananda do you think my install.sh can run in a better way? I feel a bit frustrated from running it line by line because it worked fine in the last times I checked (before creating the mistake I know fixed).

I think something in install.sh can be done better.

terdon

11:51

for one thing, what's the point of having assignments.sh as a separate file? it's only 4 lines!

for another, each time you run this it will re-add the same lines to bashrc

So if you run it 3 times, then, each time you open a terminal, it will re-source all those scripts 3 times. The more you do it, the more your system will slow down.

user9303970

Yes, you are right about assignments.sh. Seems I went too-modular there. Sorry for not mentioning I used to run it with curl but I'm having many problems running that remote script with curl.

terdon

To be honest, the entire thing is too modular. You have I don't know how many separate scripts sourcing each other and setting up variables. That just makes it very very hard to follow and understand and introduces a whole level of complexity you don't need.

I especially don't understand the point of setting up aliases that you only use inside the same script. Just use the command directly, or save it in a variable or something. Aliases are useful when doing things interactively, what's the benefit of using them in a script?

Or will you be sourcing the assignments.sh file from your terminal at other times as well?

user9303970

@terdon I no longer use assignments.sh (deleted it after reading what Kusalananda said. I know do the source form install.sh.

About aliases in scripts --- I know it is not possible, I don't do that... Where did you see me run an alias in a script?..

terdon

@user9303970 github.com/benqzq/uncwe/blob/master/templates/aliases.sh

user9303970

I just declared them for later use as I generally use all of them interactively.

terdon

12:00

OK, but then you have things like this:

#!/bin/bash
cat ~/${repo}/templates/nginx_default > ${s_a}/default

That's your uncwe/conf/nginx.sh

user9303970

Why is it bad? This saves me of changing the default Nginx file by hand each time.

terdon

So you have a separate script that runs one command, why? And you are using two separate variables that aren't defined there, so you have no way of knowing where they are defined, where they come from, what happens if either of them isn't set or is set to a bad value?

You also have most (all?) of your variables unquoted which is a very bad idea.

205

Q: Why does my shell script choke on whitespace or other special characters?

Or, an introductory guide to robust filename handling and other string passing in shell scripts. I wrote a shell script which works well most of the time. But it chokes on some inputs (e.g. on some file names). I encountered a problem such as the following: I have a file name containing a spa...

user9303970

They are exported in the variables file that gets exported each time I start using the OS.

terdon

148

Q: Security implications of forgetting to quote a variable in bash/POSIX shells

If you've been following unix.stackexchange.com for a while, you should hopefully know by now that leaving a variable unquoted in list context (as in echo $var) in Bourne/POSIX shells (zsh being the exception) has a very special meaning and shouldn't be done unless you have a very good reason to....

bash shell shell-script security quoting

@user9303970 Yes, but when you come back to this in 3 months time, you won't remember and will have to track everything down again.

Wait, each time you start the OS?

How?

user9303970

Oh, you mean to ${myVar} instead "$myVar"?

terdon

12:04

@user9303970 Yes. Or "${myVar}" if you like, but never $myVar or ${myVar} unquoted.

user9303970

I believe I'll remember, honestly... This is just a small personal environment I maintain regularly (as you said, it's best for personal usage and not to be shared with anyone, I'll usually work differently for a non-personal project).

Each time I start the OS because the variables file is sourced in ~/.profile.

terdon

@user9303970 Even for personal stuff, you really want to get into the habit of always quoting your variables. I strongly urge you to read the two questions I pasted above.

@user9303970 Then it is read each time you log in, not each time the OS is started.

user9303970

Oh there is a misunderstanding here @terdon I do intend to quote them. And thanks!

Oh sure, each time I login.

Sorry...

terdon

Anyway, the single most important thing you really must fix is the -y option for apt. That is an awful idea. It means it will silently say yes to any prompt and you absolutely don't want that when installing things like this since it can lead to a broken system.

Think "Package A conflicts with packageB, remove packageB?" and you say yes automatically but need packageB.

user9303970

@terdon I install an extremely small number of well commnual packages...

terdon

12:09

Oh, also, these lines:

user9303970

Why should I not automate that process?...

terdon

@user9303970 Yeah. That's what we all think until one of them breaks our system.

@user9303970 Because things change. And you are using PPAs which can stop being maintained at any time.

And these lines then:

printf "\n%s" "source ~/${repo}/templates/variables.sh" >> ~/.bashrc
printf "\n%s" "source ~/${repo}/templates/aliases.sh" >> ~/.bashrc

Apart from being added over and over to bashrc each time you run this, they will result in a bashrc that doesn't end with a newline. You want printf "%s\n" not "\n%s"

user9303970

Only certbot, I don't plan to use any other PPA besides that.

I run install.sh only ocne.

once*

I'll fix this.

Only certbot, I don't plan to use any other PPA besides that, on this machine*

terdon

You could also do something like grep "source.*variables.sh" ~/.bashrc || printf "%s\n" "source ~/${repo}/templates/variables.sh" >> ~/.bashrc

That way at least it will only be added if it isn't there.

@user9303970 So? What happens if certbot causes a conflict? Package conflicts are a common problem.

user9303970

I assume that I'll just recreate it's installation script or use different software.

terdon

12:24

No, with the -y option it will just do whatever apt thinks is best which isn't always a very good solution.

user9303970

12:41

@terdon Okay, I'll see what I'll do then, in such case. Maybe I'll use Ansible by then. Either way, it's unlikely with Certbot which is the only and will 99% keep being be the only PPA I'll install here. Thankfully a VPS rebuild is a matter of a few seconds (I also usually have external weekly backups and I also tend to make immediate ones).

Again I emphasize that I'll usually not conduct this way with any other PPA and especially in a non-personal project and I agree with the notion that when working with PPAs, -y is usually should be avoided.

Kusalananda

(sorry for not being able to take part in the discussion, I'm busy with work)

Jesse_b

17:36

is there any way to put a backtick in an inline code block on here?

terdon

@Jesse_b Chat or main?

The answer is yes to both, but it's a pain.

Which is yet another good reason to use $(command) instead of ` command `

Jesse_b

in an answer

0

A: BASH subprocess using SSH in script?

Your command substitution is expanding locally because the double quotes allow the shell to. The double quote will allow the shell to interpret $, ```, \, and if history expansion is enabled ! Single quotes on the other hand will preserve the literal value of all special characters. So when...

I need to specifically mention the back tick

terdon

@Jesse_b like this:

a `` `foo` `` b

two backtics, then the one you want to display

Jesse_b

ty sir!

terdon

I never remember. I always have to fiddle around with them too.

But Michael has a post on this on meta somewhere

Ah, even better:

154

A: How can the backtick character ` be included in code?

If you do not want to use a pre-formatted block, there is still a way to do it inline. From the “Code” section of the Markdown Documentation: To include a literal backtick character within a code span, you can use multiple backticks as the opening and closing delimiters: ``There is a litera...

Jesse_b

17:46

ibb.co/mZWCAn

terdon

snort

JBallin

hey I'm trying to enable syntax highlighting in nano so I added the following to ~/.nanorc:
include "FILEPATH/javascript.nanorc"
but when I source the file I get:
bash: include: command not found

Jesse_b

I don't think you source nanorc?

Have you tried just opening nano

Faheem Mitha

Yes, nano would just read its config file.

JBallin

ah, thanks!

works now

Faheem Mitha

17:57

You can't source an rc file - it's not a shell script.

Jesse_b

:-)

bashrc gets sourced

Faheem Mitha

@Jesse_b Is .bashrc is a shell script? I suppose it is.

Jesse_b

@FaheemMitha Well a file doesn't need to be a script to be sourced. I use some source files that only have functions/variables in them but don't actually execute anything.

Faheem Mitha

@Jesse_b Is something only a script if it executes something?

Jesse_b

@FaheemMitha I suppose that question is over my head

Faheem Mitha

18:01

@Jesse_b Unlikely.

Jesse_b

Well in my opinion a script would be something that executes

Faheem Mitha

@Jesse_b ok

Jesse_b

The files I have I refer to as config files, they don't have a hashbang

Faheem Mitha

@Jesse_b They still contain shell commands, though.

Jesse_b

@FaheemMitha Yea I suppose it could very well be argued they are shell scripts as they are a series of commands but the execute part could be debated I think

A script or scripting language is a computer language with a series of commands within a file that is capable of being executed without being compiled.

is assigning a variable considered execution?

Faheem Mitha

18:09

@Jesse_b Dunno. Sounds like the kind of insanely trivial question that might get upvotes on U&L.

Jesse_b

I like to debate things like this but I don't like how it often leads to real arguments and upset feelings

Faheem Mitha

@Jesse_b Upset feelings over nothing? I hope not.

It's strictly a matter of semantics. I'm tempted to post about it- I think there's a good chance that it would be swiftly closed.

Jesse_b

@FaheemMitha It may be opinionated but I'm not sure. I think there may be a clear answer out there somewhere

derobert

@Jesse_b errr, yes?

Faheem Mitha

@Jesse_b there you go.

derobert

18:16

:-)

Jesse_b

@derobert So bashrc is a script and sourcing a file full of variables is actually executing a script?

Faheem Mitha

@derobert So is a shell rc file a shell script then?

derobert

yes

Faheem Mitha

Excellent. Issue settled. (Yay.)

Jesse_b

:)

derobert

18:17

I mean, it's a bunch of lines of code being fed to a command interpreter. As long as you consider running a script of any sort execution...

Jesse_b

Well you can also use command substitution in a source file and that is certainly execution

derobert

Aha! Found out why this is:

time apt-cache policy bash
bash:
  Installed: 4.3-11+deb8u1
  Candidate: 4.3-11+deb8u1
  Version table:
 *** 4.3-11+deb8u1 0
        500 http.us.debian.org/debian jessie/main amd64 Packages
        100 /var/lib/dpkg/status

real    0m56.825s
user    0m28.580s
sys     0m28.144s

... apt-cache policy bash should not take a minute.

Fix seems to be... ulimit -n 1024 :-/

as apt seems to go through every possible file descriptor and set close-on-exec on it.

Jesse_b

On the plus side apt-cache policy bash is really fun to say

derobert

apt-cache policy BASH!

Jesse_b

what is the difference between ulimit and ulimit -n

derobert

18:23

-n makes it set the open files limit

Jesse_b

➔ ulimit
unlimited
➔ ulimit -n
256

oh so ulimit is fds and -n is files

derobert

without any options is probably vss (memory)

you can do ulimit -a to display all the limits

or help ulimit (at least in bash) ought to give an explanation

possibly this used to be fast enough to go through 1M fds, but that's a very syscall-heavy load, so Meltdown mitigations may have really hurt. Or maybe a Docker or image upgrade upped the fd limit. Who knows. Easy enough to fix though!

Jesse_b

I need to learn docker. It's a bane of my existence

Faheem Mitha

Should one leave the ulimits as default values?

derobert

@FaheemMitha depends on what you're doing.

they can be quite useful to keep a runaway program from bringing the system to its knees.

Or sometimes you have to up the fd limit for some servers (that need to have more then 1024 file descriptors, which is often the default)

Faheem Mitha

18:30

@derobert Well, under normal circumstances when you're not doing anything in particular.

derobert

No, then normally the defaults are fine.

Faheem Mitha

@derobert Ok, thanks.

derobert

Yeah. Getting to rebuild my base containers... If I recall correctly a git push will make that happen :-)

@Jesse_b Docker for GitLab's CI stuff is really useful.

Jesse_b

@derobert I'm not very well versed on it. I just know people abuse it in my companies cloud

user9303970

19:15

I deleted some files, merged some files, renamed dirs and some files and got a much neater Bash program. I hope someone wants to review my Ubuntu-Nginx-WordPress bootstrapper again.

github.com/benqzq/uncwe

αғsнιη

Hi

anybody knows how do I do ftp without specifying username/password within shell script? I didn't find any related posts on SE sites

derobert

19:53

@αғsнιη .netrc ?

Kusalananda

@JeffSchaller This post also had a dangling word ("favourite") at the start of the text before I removed it. unix.stackexchange.com/questions/423681/internal-calls-problem

It's a cross-post

Kusalananda

20:15

@αғsнιη Yes, as @derobert says, the .netrc file is for things like that. It may have its own manual on you system (man netrc, it works on Ubuntu).

Kusalananda

20:52

Wow, a new definition of the word portable: "works on any Linux"

:-)

2

Q: What is the portable way to get the default route source address?

I need to get the IP address used as the source for packets sent via the default route. My first idea was to use ip r and deduce this from there: # ip r 0.0.0.0/1 via 10.0.2.1 dev tun0 default via 10.237.76.1 dev enxb827eb4297a4 src 10.237.77.206 metric 202 10.0.2.0/24 dev tun0 proto kernel sco...

linux networking ip route

Kusalananda

21:49

@Jesse_b Not in terms of "executing an external utility", but "variable assignment" is a type of "simple command".

Jesse_b

@Kusalananda Yeah, I'm definitely convinced now that I was wrong but I still will refer to my files as config files rather than scripts to avoid confusion :p

Kusalananda

Huh, I accidentally typed man g?? (instead of man g++) on macOS and got the manual for cp.

Jesse_b

but even the man page on the source command says execute

{0} ➔ man g
No manual entry for g

Do you have an alias?

Kusalananda

@Jesse_b Well source does cause the commands in the file to be executed.

Jesse_b

@Kusalananda Yeah, I mean I don't want the files to be confused for scripts and executed rather than sourced

Kusalananda

21:53

@Jesse_b No alias. And on my OpenBSD machine, it brings up g++(1)...

@Jesse_b Not g, g??.

With the questionmarks.

Jesse_b

ah yea that opened the cp man page for me too

Kusalananda

Wonder why...

Jesse_b

if you man ?? it opens the man page for pv

oh I know

man gcp

still seems arbitrary. I would think man ?? would open cp(1) before pv(1)

also what is the gcp gmv grm about if they are BSD tools and not gnu

Hmm well nvm they are gnu

TIL: I can add g to the beginning of commands on my mac to use the gnu versions

Kusalananda

@Jesse_b Yeah, I have coreutils installed under /usr/local and they all have g prefixes.

Jesse_b

@Kusalananda Have you manually installed GNU coreutils? I think I may have at one point which may be the reason I have them

Kusalananda

22:09

It's a port in OpenBSD (sysutils/coreutils: openports.se/sysutils/coreutils)

I keep it around to be able to answer Linux questions ;-)

I've figured out what the man ?? thing is. It's applying ?? as a filename globbing pattern against the available manuals. The first matching one will be shown.

Jesse_b

yea

?
Matches any single character.

Kusalananda

man ???? shows the manual for arch(1) on my machine.

Jesse_b

mine shows the manual for node :p

Kusalananda

But still... g??... --> cp

Jesse_b

it's not

scroll down

it's the gcp manual

Kusalananda

22:13

Duh!

+15 rep to you :-)

Jesse_b

I'm trying to figure out how it picks the globbing order now lol

duh, PATH

Kusalananda

Probably lexiographically.

Ah, on the path. Yes.

Jesse_b

That's why man ???? gives me the node man page

/Users/jessebutryn/.nvm/versions/node/v6.11.5/bin:/usr/local/bin:/usr/bin:/bin:‌/usr/sbin:/sbin:/usr/local/NOCTools:/opt/X11/bin

I need to fix my path :-/

Kusalananda

Hold on, you mean PATH, aha. Did you say you're on macOS?

Jesse_b

{0} ➔ uname -v
Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64

Kusalananda

22:21

What shell?

Jesse_b

bash v4.4

Kusalananda

From man(1) on macOS: "PATH helps determine the search path for manual page files. See SEARCH PATH FOR MANUAL PAGES."

That's interesting.

Jesse_b

Yea I thought it was that was on linux/unix as well

Kusalananda

The man search path (or MANPATH, or whatever) has always been a mystery to me. Different man implementations does things differently.

Jesse_b

oh yea

Kusalananda

22:24

On some systems (like macOS) it's enough to add something to PATH, on others you have to be more inventive.

Jesse_b

on SunOS it's MANPATH

Kusalananda

On some systems, setting MANPATH overrides the system's manpath (actually, that may be across the board on all systems)

Jesse_b

Search Path
Before searching for a given name, man constructs a list of candidate
directories and sections. It searches for name in the directories
specified by the MANPATH environment variable.

In the absence of MANPATH, man constructs its search path based upon the
PATH environment variable, primarily by substituting man for the last
component of the PATH element.

Kusalananda

With OpenBSD you have man -m somepath with will augument the system's MANPATH, which is nice.

Jesse_b

I don't know what I just pressed in my unix terminal but I hosed it >.>

Kusalananda

22:26

:-)

Jesse_b

speaking of man pages, I made this function the other day:

gman () {
    local this_command="$1"
    if ( ! which -s "$this_command" ); then
        echo "ERROR! $this_command does not exist"
    fi
    if ! man "$this_command" 2>/dev/null; then
        googler "$this_command" "man page"
    fi
}

requires googler

Kusalananda

I've seen similar suggestions, but hooking up the diagnostic output of gcc to StackOverflow.

Why is which in a subshell?

Jesse_b

I like to use subshells for tests if I can just to make it easier to read

Kusalananda

But not for the call to man?

I just saw the inconsistency there. If they had been the same, I wouldn't have mentioned it.

Jesse_b

Yes I should do it there as well. I don't know why I thought the subshell would hide the output of man

I should also remove the if constructs and just ( which -s "$this_command" ) || echo ...

Kusalananda

23:00

@Jesse_b There's a fine line between "neat" and "hard to follow" though, and that lines goes somewhere around removing the if (for me).

... at least when the commands gets too long.

Jesse_b

well in this case it would be short:

gman () {
    local this_command="$1"
    which -s "$this_command" || echo "ERROR! $this_command does not exist"
    man "$this_command" 2>/dev/null || googler "$this_command" "man page"
}

Kusalananda

23:12

Midnight, sleep time. See you later.

Jesse_b

goodnight!

Transcript for