Just the interface with whatever IBM Z's version of VT-x is, or even things like virtual device drivers?

every VM cannot touch each other unless explicitly configured to do so

That's true with nearly all hypervisors.

you can specify the cpu instructions that can be executed

With fine granularity? I know that VT-x can do that as well, but only for certain instructions.

Like in VT-x you can restrict RDTSC, but not CLFLUSH. Is it the same with z/VM?

you can encrypt memory AND storage with a key stored on the secure processor that nothing can read but that VM, so even if someone pays for the storage or vm administrator, all data is encrypted

Oh neat. So kind of like AMD's SVM?

(skimming vm.ibm.com/library/zvmsecint.pdf right now)

go to redbooks.ibm.com and get a technical manual

I helped write the latest version

oh huh

So I take it you know quite a lot about hardware virtualization.

that paper is very good... altmark knows everything

I know a lot on s390 architecture, and I carry over my knowledge over x64 as analogy, because it's not the same

I have to say, at least from reading the above link, it doesn't seem like it provides much more security-wise than KVM with AMD-v and SVM. The security techniques it describes seem rather generic.

most I know on x64 is to compare to s390 and prove that s390 is better

Anything is better than x86 (and x86_64).

x86 is a monster.

it seems so, but the difference is that security was baked on the architecture since day one

Into s390 or into z/VM?

VM was created back when companies had ONE computer and had to test their system before migrating, and wasn't even possible to think about buying another computer only for that. so they had to use the same computer for testing AND production. and how can you convince execs that your devs will use the same computer and data for testing? so VM was born with all boundaries defined on the hardware

The only thing I think x86 has going for it is some very useful security features that I presume s390 doesn't have, like SMEP, SMAP, UMIP, MPX, MPK, TME, TSME, CET, etc. But those are bolted-on. Of course, other architectures have their own versions (like ARM's PAN/PXN).

@forest both are good reads... s390 will say about the architecture and more about zOS, and the zVM one will tell about the virtualization

@ThoriumBR Ah, so the difference is that VT-x and AMD-v were designed on top of x86, whereas s390 was built specifically for virtualization, which z/VM can take advantage of more efficiently?

yep. the core of zvm uses 2MB of memory because it only cares about virtualization and nothing else. it was designed for that.

neat

If I knew more about virtualization security, I'd love to try fuzzing it.

tcpip is a guest. and it only does TCPIP and nothing else. console? config files? commands? that's not tcpip and are managed by another guest

So sort of like using Xen to isolate individual drivers as a sort of makeshift microkernel (but with privilege boundaries enforced by the hypervisor rather than by the kernel and MMU)?

kind of, because drivers share the same memory space. on zvm each driver have its own memory space.

they aren't called drivers, but the idea is mostly the same

Do the guests still have their own TCP/IP stack?

and the hipervisor isolates every guest from each other so you can run zvm inside zvm (we call it second level zvm), and that second level cannot tell if it's running on hardware or inside zvm. and you can install a 3rd level, or 4th level, and no level can tell if they are on hardware or not

You can't tell if you're on another level with nested VT-x either, until you use a side-channel attack. :P

yep, that tcp stack is the "physical network", so it's the network card of the guest... linux still have its own network

I have a feeling that a side-channel attack could fairly easily tell what level it's running at.

and the real network card is not a card, but almost an AIX running on ibm power

AIX? O_O Gross...

I've set up something like that (although on x86), by using VFIO to bind network cards.

@forest you could tell you aren't on hardware, but no idea if you are on 2nd or 4th level... performance would suffer pretty badly after 3rd level

If I understand what you're saying, it did basically the same thing.

the drivers are the same on hardware, or virtualized, so you couldn't tell

@ThoriumBR I'd think you could tell based on said performance suffering. Attempt to run an instruction that'd get trapped (whatever the equivalent of vmexit is), and see how long it takes. The longer it takes, the more levels it had to "fall through".

it's only a message queue anyway, so you cannot tell the real card is answering the queue or someone else is MitM-ing the queue for you

heh

Yeah same with a VM binding to a NIC with VFIO.

@forest and here you are mistaken... you can make your guest on the 4th level run on the physical processor... the hipervisor cuts all layers, allocates memory, and dispatches you on bare metal

so your instructions will run as fast as if native

@ThoriumBR Sure it'll run on the same processor, but when it triggers a vmexit (or whatever the equivalent is), doesn't it trap into the hypervisor which can decide whether or not to let the instruction continue unchanged or be emulated? Or is the ISA designed to handle that with more complex features than "trap if executed on a vm"?

performance will suffer because every layer will have to allocate its own memory, check bondaries, check if your guest have the correct privileges BEFORE dispatching it on the next layer

but when everything is fine, the guest runs on real hardware without anything in between

there's no emulation. it's native s390 instruction set

you can run it on hardware, you can run virtualized.

So what happens when the guest runs OUT or a similar instruction that would allow bypassing the VM? Is it not like VT-x where it gets trapped and the hypervisor emulates that (one) instruction before passing control back to the guest to run natively on bare metal?

there's nothing like that... imagine the hipervisor as the protected mode on the x86

real mode, actually

there's no instruction to bypass the hipervisor because there's no instruction to bypass the hardware

Oh, so its ISA's version of OUT would itself know whether it's in a guest or not?

And this is for s390?

yep. for the guest there are no special instructions that work on hardware but not on bare metal or vice versa

unless of course you disabled them on the guest configuration. but that's a config option, not architecture design

That's so foreign to me. It's hard for me to grasp a virtualization which doesn't trap certain instructions into the hypervisor.

so if you give all privileges to a guest, it won't know if it's hardware or not

And if it has all privileges, it'll be able to do anything it wants.

imagine you can stop the clock on the process

And I imagine the ISA itself supports this?

so you can change memory whatever you want and the process have no way to know... time didn't passed, so for the process the hardware is only his

Do modern z/VM systems run on s390 or do they use another arch now?

you copy memory back, recover the registers, resumes the clock, and the process resumes without knowing that anything changed at all

@ThoriumBR Yeah but that's how VT-x works as well, and it still has to trap instructions.

because the hipervisor owns the CPU, it can do whatever it wants and the processes running on it don't even have how to detect anything, because even the clock is outside of its world

it traps some instructions, but both on hardware and on guest, so there's no difference in the viewpoint of the guest

guest does not know if the hipervisor trapped the instruction, or the cpu did it

Hm

I guess it's too foreign to me to really understand the concepts involved. It seems like a hypervisor configuring the CPU to change the behavior of its instructions would create far more overhead than doing the traditional "set insn to trap if in guest, switch control to hypervisor to deal with insn".

I forgot one information... on s390 there's an "entity" called PR/SM that "owns" the hardware... nobody talks to the hardware but PR/SM... zVM talks to it and passes its interfaces to the guests. so the guest uses the same interfaces no matter if running on "real" hardware (talking to PR/SM) or talking to zVM

And it's the same for z/Architecture?

this is the z/arch

Hm, maybe ibm.com/support/pages/sites/default/files/inline-files/… will be helpful.

yeah, the holy book of principles of operation

that describes everything on the hardware platform

it would help if it wasn't so large

So it really does seem like it was built for virtualization from the ground up.

yep. it was its only mission

I guess x86 is replacing it, despite x86 virtualization being duct taped on.

before the s360 architecture, every new computer means throw away all your systems and write a new one from scratch. s360 meant you could buy a new computer and run your old software because the architecture was virtual, and something in between translated the virtual architecture (the s360 instruction set) to the actual hardware.

I wonder how z/arch deals with spectre-class attacks, if it does at all.

s360 turned into s370, into s390, and s390x, and you can today get a binary compiled on s360 and run native on the latest mainframe with 4tb of memory without changing anything

now do the same on a ms-dos COM file from 1986...

oh nice

Definitely seems like a superior general design compared to x86 (or even ARM).

It looks from that document like privileged instructions will raise exceptions, which implies that it would be trapped by the hypervisor as it would be with VT-x.

s390 is for a different kind of job. it was made for heavy IO and data processing, where a failure meant being sued by the client. so failure wasn't an option

a couple generations ago (ibm z900 IIRC) got the same instruction executed twice on different processors, and they compared the results. If the result was the same, the next instruction got executed. If the result differed, another processor executed the same instructions to see which one was at fault, and the faulty one was turned off

every mainframe shipped with some spare processors so they would take over any faulty one

it had error correcting memory 25 years ago, RAID on memory...

Modern high-quality fabs and extensive ECC in SRAM have made such voting-based redundancy largely irrelevant, at least in the CPU itself (ECC needed for RAM and RAID for disks, separately).

now they don't do that anymore, but 25 years ago it did. IBM noticed that execution errors were not occurring anymore and stopped doing it

ah

That sort of thing is still done in airplane computers that use fly-by-wire.

but they still can detect a processor error and can shut it down and re-execute on a spare, so you can go over the box, take out one processor with you and the application will not notice

it will pause for a few seconds to sync memory and activate the spares, but you won't lose data

That's neat. So an MCE will cause a snapshot of the execution context to be saved?

cache is shared. there are several levels of caches, so they store the history of the instructions running

so if a processor dies, the next one can load the last successfully executed instruction from cache, and resume from that

wow

and cache isn't a chip, is a subsystem with P2P communication

you know DHT? yep... but on hardware

Internal to the chipset, I assume?

Oh god, that sounds like it'd have horrible latency.

no, it's not

x86 having cache as SRAM transistor banks inside the chipset itself is quite fast.

No need to involve any kind of communication off-chip.

the CPU only executes instructions. memory access isn't his job. interruption handling? nope. cache access? nope...

That's true with x86 as well, but only because "CPU" is one small part of the chipset itself.

Modern CPUs are amazingly heterogeneous. They're huge networks with massive pipelines between different components and complex internal scheduling (If Anders Fogh taught me anything).

so when a MOV [AX], [BX] hits the CPU, the memory access chip read that instruction ages ago, and asked the cache subsystem the content of [ax]... cache subsystem looks into L1, L2, L3, L4 and L5 if someone have those values, and gets the fastest one inside L1 on the CPU. so when the CPU goes to execute the MOV, that data is inside L1 already

Yeah that's how it works on x86 as well (though it usually only has L1, L2, and sometimes L3).

for a s390 processor, memory access is considered IO and the process accessing memory is suspended and sent to the execution queue

And since it's a Harvard architecture cache, there's separate icache and dcache.

and the caches are gigantic... one chip can have 96mb of cache

if you sum all levels

and the newer mainframes can have 80 chips

And here I thought my 24 MiB of L3 was gigantic. :(

my first mainframe was a z900 with 700mhz... it was running 360gb of virtual memory using 64gb of physical memory and we could do 3000 SSL handshakes per second, and we achieved 4gb of disk IO per second

back then I had a top of line Athlon XP 2600, it was the fastest processor on the market, I had the fastest ASUS motherboard available, and the fastest Corsair memory available, and I did 3gbps of IO on RAM...

I was single, had a good paying job at IBM, and my side job was mounting and selling custom computers, so I saved one year to buy the absolute best computer I could, and a 700mhz mainframe laughed at me...

wow

it was faster doing disk IO than I could do RAM IO

I guess that goes to show that a system optimized for IO will be really good at IO!

I wonder how mainframe vs enterprise CPUs compare nowadays.

and it is... ah, and the IO bus isn't a bus... it's a ... let me guess... a system.

Like a modern z/Arch vs a high-end Xeon.

you cannot compare... you can run a mainframe on 99% CPU for weeks and the system will run nicely... run a x86 over 50% and performance suffers

I've never had the experience that performance suffers for running an x86 at high usage.

I manage systems that run at nearly 100% and if I use rt scheduling policies well, the system feels as responsive as if it's totally idle (of course, cache misses go up).

a 99% mainframe means it still have spare capacity.... a 50% xeon system means caches are invalidated faster than it can be reused, IO queue is awful, and the instruction pipelines are clogged

@ThoriumBR If timeslices are fairly long, you won't see pipelines or caches get overwhelmed like that. That's one thing the FIFO rt scheduler policy in Linux is good for.

I once put in production a 4 CPU mainframe running Oracle that was about 30% faster than a 4-core 4-box Dell Xeon running Oracle

Of course, my workload is quite different from that of a mainframe (less IO and simple request serving, more compiling, encoding, and data analysis).

and I installed a system wit 4 CPU that ran in 45 minutes a batch that took 46 hours on 20 (IIRC) xeon processors

haha what

ERP

I mean, mainframe CPUs may be well optimized, but I can't imagine that'd be the case (45 min vs 46 hours) unless the Xeons were really, really badly configured.

it managed the payment of the government for a large State here... it would take 4 days or more to calculate the paycheck for everyone before the mainframe

Was that before SIMD was common on x86?

those xeons didn't had enough IO bandwidth to ingest all data fast enough

Oh, so IO was the limiting factor, not the actual fetch/decode/execute in the CPU.

it was on 2007, so x86 arch was ancient

Ah

I'm sure SSE/AVX greatly improves all of that, but it wouldn't help with poor IO.

and the system was strange: it would load data from oracle, generate C code, compile, and execute that C code later

wot

That is strange.

the first time it ran, people thought the mainframe aborted execution... so they compared the resulting tables with the test case and it was correct... so they dropped the database, imported everything back, ran again. 45 minutes, all correct.

the month it went into production the largest newspaper had on the first page: "For the first time ever, public sector pays its employees on the 1st day of the month"

I may have the screenshot saved somewhere

even today, a well configured mainframe beats an Oracle Exadata easily

Nice job!

I must be seriously underestimating just how specialized the jobs of mainframes are.

my last oracle PoC beat an exadata twice its size and a generation newer by 25%

it's a nice job but a dying one

Because enterprise x86 processors are catching up, or because experts are dying out?

IBM policies are terrible, licensing is draconian, it's close to impossible to learn zVM unless you work for IBM or a large company that owns a mainframe, there's no "student license", there's no cheap emulator (the one that exists costs 5k or more, and under a hefty NDA)...

you can call marist college and ask for a zvm guest for the community cloud, but almost nobody knows that, and IBM does not publicizes it

Ah...

and vmware/ms/amazon and so are marketing cloud as a new thing, and zvm does cloud since before it was cool but does not talk about it, so when you say "virtual machines" you imagine vmware, but zvm does it for 53 years

Yeah I'm guilty of that too. I only think of KVM, Xen, HyperV, and ESXi. I was aware that IBM (and Sun) had some virtualization stuff long before, but I had no idea it was anything but historical.

I can spin up a full VM with disk, IP, custom software and whatever in 5 seconds... not a container, a full VM

and there's an API for everything since always

I can clone a 64GB disk in 0.1 second, I can clone a running VM in 0.1 second and give it to you

To be fair, modern KVM systems with well-configured QEMU can do that as well.

Since you can use CoW for cloning disks of arbitrary sizes, and spinning up a VM takes only as long as the guest running inside takes to initialize (or less if it's migrated from a snapshot).

yep, but zvm does this since 2000's, but nobody knows

That's surprising. I guess IBM management is really bad.

if you think windows xp is newer than that, that's something...

do you know linux on mainframe was a not authorized project?

not a "hobby project", but a "stop this right now" project

What do you mean?

Linux on Z?

yep... engineers were porting linux to s390, manager got to know, ordered the project to stop and the code to be destroyed. engineers "agreed" and secretly resumed the port until another manager got to know and let them resume

haha wow

opensource.com/article/19/9/linux-mainframes-part-1

That's hilarious.

I guess Linux really does get ported to everything.

(I am a bit irritated that the Wikipedia article on mainframes claims the OSes they run are more secure because there are less CVEs for them than Linux/Windows/Unix...)

it's technically correct... a helicopter is safer because less thieves know how to hotwire one

and a manual transmission is safer on the US because less thieves know how to drive one

and a mainframe is safer because just a handful of people know how to write shellcode on s390

But the thieves you have to worry about are often the ones who will learn how to do it before they exploit it. So it's not safer. It's security through obscurity.

and as far as I know, metasploit does not have modules for s390

The best hackers don't use metasploit for serious campaigns. :P

you are correct... random guys won't hit a mainframe, but dedicated ones will

A sophisticated attacker will learn the system and will find vulnerabilities, and then the only thing that matters is how hard it is to find vulns, not how many easy-to-use tools exist for the platform.

And in that situation, I'd wager that mainframe OSes are quite insecure at their core.

Even if the hardware itself is designed with security and partitioning in mind.

but if you have to buy the hardware for a million bucks to train, the pool of hackers available to hack one is pretty small

It's bigger than you'd think.

@forest they aren't... IBM have a dedicated team trying to break everything all the time...

The pool of hackers who know right now how to do it is small, but the pool of those able to learn how, without buying the hardware, is actually quite large.

@ThoriumBR As do nearly all major projects.

Microsoft and Google are probably the best in that regard.

and ibm software is terrible, but hardware is anothe level

and for IBM, the OS counts as hardware (go figure)

Do IBM OSes even have any equivalent to SMAP/PAN (denying access of userspace memory while in kernelspace)? I'd wager that the most they have is NX.

on zOS, every process get its own address space without any way to read anything else

on zVM I have no idea

That security is the bare minimum. No system with an MMU today isn't like that.

but it have EAL 5+ certification, so the memory isolation is the same as separated computers without communication channels between them

EAL5+ doesn't mean very much for operating systems. RHEL has a high EAL certification and it's still Linux, with the same vulnerable syscalls as all the others.

All EAL does (unless EAL6/EAL7) is ensure that the specifications are sufficiently secure, not that vulnerabilities that violate the specifications do not exist.

Hell, Windows XP has EAL4! And that's with the infamously-insecure SP1.

for zvm it means guests running under them cannot interfere with each other

Assuming no bugs in the hardware (although admittedly it's likely that the hardware is secure). But I tend to doubt that the OS itself is particularly secure (z/OS).

EAL certification is very often misunderstood to mean proven correctness, when only EAL7 does that.

What EAL5 means (for example) is that there is no way via the legal APIs to violate certain security guarantees, not that there aren't ways to break the APIs. EAL5 doesn't even attempt to check that.

Solaris got EAL4, and it has a kernel even less secure than Linux.

And just because z/Architecture's LPAR (looks like that is the analogous term to VT-x) is EAL5 doesn't mean that all the syscalls you can call from within z/OS are secure.

Looks like z/OS and z/VM are EAL4, like RHEL and Oracle Linux.

(You can tell I'm an extreme skeptic of OS-level security! :P)

I know, that's why you have a security manager on top of that

Security manager?

yep: RACF

Resource Access and Control Facility

Is that something which configures LPAR policies?

it manages every resource inside zOS

or inside zVM

ah

But it runs in kernelmode, right?

yes, it runs on kernel mode... but the guests runs on user mode, and RACF manages resource access

Which means that it only provides security so long as there are no vulnerable syscalls.

so even if there's an API violation, RACF can see that violation and kill the VM

Oh, this happens outside the VM/outside z/OS?

So if the VM attempts to, for example, write to a virtual disk it doesn't have access to, RACF will kill it?

if you design the syscalls on a way that isn't possible to data on a guest be read from another guest, isn't a problem

Only if your threat model is exclusively about guest-to-guest attacks.

But if you have a malicious process A within a single VM and it wants to attack process B in the same VM, RACF and LPAR won't protect it. A local malicious process will still be able to take over that VM if it exploits its kernel without violating RACF. It may prevent VM A from attacking VM B, but not process A in VM A from attacking process B in VM B.

@forest it's not possible to do so. a VM will have disks 100 and 200, for example. trying to read/write to disk 300 will not work because that disk does not exist on its universe

@forest it won't, because zVM job is create a secure environment to run guests... inside the guest they don't touch anything, it's guest's problem

@ThoriumBR I was talking about z/OS. I'm sure z/VM as a hypervisor has a small enough attack surface area that it's reasonably secure.

RACF will make impossible to Guest A to even know Guest B exists

Well, sans microarchitectural attacks, at least. :P

It sounds like RACF is the equivalent to, say, libvirt on RHEL.

and even if the admin knows about guests A and B, it cannot reference anything from one guest on the other unless the channel is configured so they can share data

I'm sure z/Architecture is reasonably good (or even very good) at ensuring that VMs have the same security guarantees as if they were running on separate computers.

@forest it's very small, and security is the main attribute

z/OS is? It may be small, but kernels are in general really hard to secure.

And people tend to rely too much on a secure specification and not a secure implementation.

so every time a new function is proposed, they have to ask if this new function could possibly bring a security issue... if so, they don't implement until they are reasonably safe that security cannot be compromised

What do you mean by new function? Syscall? libc (or whatever) function?

for example, a virtual switch took a couple generations of zvm to be implemented because they had to test every single way data from a guest could end on the other guest

But that doesn't say anything about the security of the z/OS kernel. Just because the architecture securely implements virtualization in a way that effectively isolates guests doesn't mean that an individual guest can be protected from maliciously formed input.

it's something basic, but it could create a security issue, so no basic fast networking on layer 2 until it was deemed safe

guests cannot be protected and that's not the idea... the idea is to protect the hipervisor and the other guests against a malicious guest

and that's not the idea - But it should be. The team that develops z/OS should ensure that one process within the VM can't attack another process within the VM.

if you have a malicious guest, it will harm itself and nobody else, that's the principle, and that's the very reason vm was created

so companies could have test environment on the same hardware as the production environment and not worry

But imagine there is no hypervisor and each VM runs on its own computer (i.e. assume the hypervisor is perfect). That doesn't mean z/OS's security has been solved.

@forest they do, they protect guests against each one, but not a guest against itself or against the hipervisor

Does the guest completely lack any self-protection? Like, does every process running in userspace on a given z/OS VM have the same privileges as the kernel on that same z/OS VM?

for the guest perspective, they are on bare metal

but zvm creates copies of that bare metal for each guest

and each copy does not have any idea that there are more copies

Sure. I'm not doubting the security of that isolation.

But as a UNIX-security skeptic, I find it hard to believe that z/OS itself is secure. Or is that not what you were saying?

the OS is not impregnable, but the attack surface FROM the guest is so small that can be deemed secure

Ah

Yeah I could see that.

there are very few syscalls from the guest that can change anything on the hipervisor, and those few are VERY well guarded

But when I put my blackhat on, I'm thinking "how do I persistently compromise a VM" first, and "how do I move laterally into another VM" second. Because if that VM contains sensitive information and I can break into it, I can exfiltrate that data pretty easily.

@forest you can own a VM easily, it's almost exclusively linux after all

It is? I thought it was z/OS.

if you manage to own a ssh privkey you can ssh into other guests because there's networking between them all (most of the time)

nope... zOS is another beast and almost never run under zVM, unless it's for testing

Oh. Well that explains my confusion.

I was under the impression that z/OS ran under z/VM, which ran on z/Architecture.

And any applications (databases, etc.) would run as a process within z/OS.

the problem is going from linux to zVM... unless you have a password for zVM and the network was badly configured, there aren't even a path for that

nope... zArch is bellow zvm, zvm runs hundreds of linux guests, and linux is linux

./configure && make && make install as always

I see. So z/OS is completely orthogonal to all of this?

zArch, zOS and specific zOS applications running

and you can install zVM and put zOS inside it alongside linux guests

ah

Yeah this whole time I misunderstood where z/OS came into play. :P

but zOS is a monster with a dozen heads and a hundred arms

I don't work with zos but have training on it, on the technical side... no much practical experience

all I know is that 499 of the fortune 500 run zOS for their mission critical applications, and the only one that does not run zOS is HP because IBM and HP are sworn enemies... so much that our hardware director greeted us not with "good morning" but "have you killed HP today?"

haha wow

I gotta go... dinner time.

later o/

Thanks for the convo. I learned a lot!

it was a good chat. if you want to know more about zvm, you can gmail me

I can manage to give you access to a zvm, I think...