Is Project Active?

[RussCannon]

Is this project in active development? If so, is it serious work or, say, hobbyist type? Just wondering. I have not been pleased with the direction the KSH2020 project has been going, and it seems to have petered out anyway. I am just wondering where this project is going and what plans are in works.

Cheers,
Russ

[marcastel]

@RussCannon in short yes, the project is active but still in early organisation phases. And all helping hands are welcome :-)

The motivation, on the short term is not necessarily to bring in new features, but to make sure that the KornShell and its command interpreter remain part of all POSIX UNIX and Linux distributions. We want to ensure that it builds smoothly for packagers and to fix essential bugs.

As this is not a funded project, participation is mostly a best effort. We have not yet laid down a roadmap or anything of the sort, as we are still a very small team. The late worldwide situation has somewhat reorganised some of our priorities; but rest assured that this is NOT a dead initiative.

[hyenias]

100% agree with @marcastel. @RussCannon, I have elected to support this ksh-community's efforts. I have been learning so much from the team over the last couple of months. Yes, our ksh-community's version of the Korn Shell is in active development. Currently, I have been working on fixing rounding issues with floating points, see #6.

[saper]

still in early organisation phase

Is there some group or organisation behind it discussing things offline? or is just about pull requests we put in here...

What kind of organisation do you think is needed?

[marcastel]

@saper by organisation I essentially meant a dedicated group of people with a particular purpose : the KornShell. We need to learn how to work together, and organise ourselves to distribute work and be proactive to issues, ensure stability across platforms, further document, and so much more!

Obviously we need some sort of communication channel and more formal processes; the later are underway with, for instance, working instructions for the life cycle management of the repository. Please bare with us as this gets... humm... organised :-)

[McDutchie]

Well, I think that getting a proper bugfix release out there is now urgent, or the world will simply give up on ksh. Meanwhile there has not been a single commit to this repo in more than three months.

The last stable 2012 release is the least buggy one available, but still has many serious bugs. I've been following the work done on the abandoned ksh2020 branch, and while it certainly had fatal flaws, there was also a lot of good work done: actual fixes of longstanding bugs that can be backported to the stable release. As my understanding of the codebase was growing, I also figured out a number of fixes myself.

So, I got sick of waiting, and went to work. My 93u+m branch is now about three weeks old and under active development. It already has a number of significant bugfixes, all of which are properly documented with rationales in the git log. I got the horribly broken regression test suite into a working state, so everyone can verify the fixes don't cause breakage. I'm also testing everything against the regression test suite of my cross-platform shell library, modernish, which has been helping me discover many corner case bugs in all the POSIX shells for a number of years now.

Unlike the ksh2020 developers, I work incrementally. I don't believe in massive sweeping overhauls, I try not to change anything I don't understand, and I'm scrupulous about not causing regressions.

The 93u+m branch/fork is here: https://github.com/modernish/ksh

This is not intended as a hostile fork. When your community gets started, I would be more than happy to contribute my fixes back (as long as it doesn't require jumping through a bunch of hoops, at least; in that case feel free to take them yourself :-)). And I hope the need for my fork will eventually disappear as you get going.

Meanwhile, if anyone else is impatient, contributions to the 93u+m branch are more than welcome. There is no organisation. Let's just use github and get to work.

[kdudka]

@McDutchie Your fork looks indeed promising. Thank you very much for working on it!

[jelmd]

@McDutchie We are all very busy and founding an org which does not operate in a try and error manner/chaotic way takes some time. IMHO it is better to wait for some task done by other members (even if it takes a little bit longer), than to merge stuff without any policy/reviews according to its own taste. ksh2020 branch has shown, that this leads to unstable, unusable SW.

IMHO, there is no reason to "push" a new release just because there was none for the last 8 years. Indeed, I prefer to release when ready. Ready means in general tested by some people for a certain time (beta) w/o problems/regressions. Right now to get "Ready" for the next milestone (beta) the plan is to review all known vendor patches, aggregate and merge if appropriate. I think that it should be doable to get a release candidate this autumn and possibly a new release around x-mas.

Any contributor is welcome and forking ksh-community/ksh master, creating a branch for a fix/problem/feature one wanna work on and when ready creating a new pull request is the right thing to do - see also https://github.com/ksh-community/docs/blob/master/workflow-HowTo.md .

So thanks for your work - I'll definitely have a look at it when reviewing the patches from apple. But that takes time and at least I'm still very busy wrt. stuff around covid19 ... However, if you intent to become a long term contributor, you may join the the ksh-community, create new discussion teams, review PRs or ask to get invited to already existing teams, where we usually discuss all the organizational stuff, which is not source code/doc relevant.

[jghub]

@McDutchie
same from here: happy to see what you are doing. you are of course right, that ksh-community has had next to no world-visible activity (prior to the corona-mess there was quite some discussion internally, though) and that things should get moving faster ASAP. so AFAIAC, it is good that you have not waited any longer.

some of the other (few) guys formally constituting ksh-community so far will probably respond separately, but I feel the bottleneck so far was that @jelmd has had not enough spare time recently to really start with integration of existing patch/bug fixes (he is the guy in this small group who volunteered for this and probably is most competent to do it in my view -- personally, I'd rather avoid to take that over, e.g.). and everybody else was obviously also very busy with other things (pandemic etc....)

regarding how to proceed: you of course might also contemplate to simply join ksh-community if you are interested (whose name is not an accident...) and drive things forward from there rather than in a fork. I am sure no one would object since from everything I have seen in the ast repo (and now in your fork) your approach sure is compatible with ours (and you confirm that in your statement above). just my 2c.

[saper]

Meanwhile, if anyone else is impatient, contributions to the 93u+m branch are more than welcome. There is no organisation. Let's just use github and get to work.

I saw some fixes from @lkoutsofios in the master. I think we could also incorporate changes to the INIT system from the 93v (which in this case is better). @McDutchie Would you be interested in accepting some patches to INIT and maybe tagging a new "INIT" release out (2020-06-XX)?

I have also noticed that setting CCFLAGS in the package breaks shared library building because ast makefiles try to set the CCFLAGS on their own.

I am also working to get the debug=1 to build again

[jelmd]

@saper As git log says: @lkoutsofios patches <= 2020-02-14 are already included in this repo. The only 2 missing patches are the ones from 2020-02-24. IIRC a free?BSD maintainer commented, that those changes are a noop because it uses something different anyway. For completeness they'll get merged, but obviously there is not really a need to give it a high prio tag.

[McDutchie]

Thanks @saper, yes please, I would absolutely be very interested in accepting patches to INIT. I've currently got it building on macOS, FreeBSD, OpenBSD, and Linux. But it doesn't currently build on (at least) DragonflyBSD, NetBSD, Solaris, AIX, HP-UX, QNX – usually because iffe fails to detect some DLL feature so we get error: unknown type name 'Dllscan_t' and other errors like it.

As for the fixes from lkoutsofios in the ast master, there was just the one fix dated after ksh-community forked from ast: a FreeBSD build fix. So I've reverted the FreeBSD fixes I ported from FreeBSD ports (which are more drastic) and applied the changes from lkoutsofios instead. Thanks for the heads-up about that. That made it stop building on DragonflyBSD though, but maybe your patches to INIT will fix that :)

[McDutchie]

Thanks @jelmd and @jghub for your kind responses. I've found the pandemic affects people in different ways, and I do have some time on my hands now. That is not going to last, though, so I need to apply the fixes that I can apply soon, or it may not happen later. I just need that free rein for now.

But to be honest, that's not all. Since I'm here I might as well be frank and put my cards on the table, so you know where you have me. So please take what follows in that spirit…

It looks like we have different viewpoints on the cause of the ksh2020 project crash. I don't think that was because they weren't acting as a formal organisation; I think it was mainly because (1) they made an initial mistake in basing their work off the incomplete and badly broken 93v- beta, (2) because they were too eager to refactor All The Things™ and not eager enough to fix bugs, and (3) because krader1961 (who was mostly driving it in practice) had an apparent disdain for the POSIX shell language that clouded his judgment in some ways. Organisations are perfectly capable of having flaws like that, and often do. Conversely, there are also highly successful projects, such as bash, that are led and driven by one person.

As I've said, there was a lot of good work done on ksh2020 as well. I've already backported a number of fixes I could not have figured out on my own, and more are waiting. krader1961 might not have much love for the POSIX shell, but he and siteshwar are much better C programmers than I am. I think the way AT&T ejected them after your campaign against them is simply disgraceful unfortunate. They did invest years of their life into this project, and without their work I would never have been able to do what I am doing now. So I am grateful to them, and I wish the situation could have been fixed without killing their appetite to work on ksh forever.

So perhaps you can see why I am a little reluctant to join your organisation. It's not only that I don't have time for the slowness, politics, interpersonal stuff, etc. that are inevitable when founding and running an organisation; it's also that you've already shown how you deal with people you disagree with. Having said that, I wish you well, and I hope you'll have great success. I would be very happy to collaborate on a more informal basis, and I remain absolutely willing to contribute all my work back whenever you're ready to accept it.

[McDutchie]

A note regarding vendor patches, which @jelmd mentioned. I have looked into some of them myself and found them mostly useless, as many of them are OS-specific hacks that make ksh build on that system while breaking things for others. If @saper can fix the build system, that would be much better.

Apple's patches in particular are actively harmful. I've no idea what they really do as there are no comments or documentation, but after I fixed the regression test suite, I found that they cause several regression test failures in things like read -n3 and others I can't remember right now. What was needed to get ksh to build on macOS were just the compiler flags from Apple's Makefile. I've integrated those into the cc.darwin wrapper script. Between that and krader1961's fix for the invalid use of memccpy(3) in sfputr(), all the regression tests now pass cleanly on the Mac.

Some of the patches from the Red Hat source RPM (which siteshwar had a hand in, of course) look useful, I'm looking into them right now. Unfortunately there is no documentation on what they do. Does anyone know where to find that?

[jghub]

@McDutchie

Thanks @jelmd and @jghub for your kind responses. I've found the pandemic affects people in different ways, and I do have some time on my hands now. That is not going to last, though, so I need to apply the fixes that I can apply soon, or it may not happen later. I just need that free rein for now.

thanks for bothering to explain your POV. I have these comments:

It looks like we have different viewpoints on the cause of the ksh2020 project crash. I don't think that was because they weren't acting as a formal organisation; I think it was mainly because (1) they made an initial mistake in basing their work off the incomplete and badly broken 93v- beta,

agreed regarding bad starting point. regarding "formal organisation": they quite obviously wanted to sail under the ATT flag to make their work "official". in my view this was de facto a counterfeit operation. regarding quality/success of their work, agreed that one does not need a formal organisation to achieve good results. but you need to be qualified to do it.

(2) because they were too eager to refactor All The Things™ and not eager enough to fix bugs, and (3) because krader1961 (who was mostly driving it in practice) had an apparent disdain for the POSIX shell language that clouded his judgment in some ways. Organisations are perfectly capable of having flaws like that, and often do. Conversely, there are also highly successful projects, such as bash, that are led and driven by one person.

agreed regarding (2). partly agreed regarding (3) -- regarding krader you probably know my position.: his dislike of the shell language sure only partly explains what has happened in ksh2020.

organisation vs. no organisation seems a non-issue to me, anyway. ksh-community technically is a "github organisation" (I believe that's the official term), otherwise it is a modest attempt of currently ≈5 people to keep ksh93 alive and hopefully in the long run provide a trustworthy upstream for package maintainers.

As I've said, there was a lot of good work done on ksh2020 as well. I've already backported a number of fixes I could not have figured out on my own, and more are waiting. krader1961 might not have much love for the POSIX shell, but he and siteshwar are much better C programmers

I think, whatever working and solid fixes are in the ksh2020 code that apply to ksh93u+ of course should be backported in due time.

than I am. I think the way AT&T ejected them after your campaign against them is simply disgraceful. They did invest years of their life into this project, and without their work I would

well, here I have to clearly object against your assessment:

1. there never was a "campaign" (which by definition would imply a concerted effort of a group of people to achieve a certain goal by probably questionable means). I am rather sure you have followed the story/history so I am a bit astonished that you arrive at this assessment. for newcomers to this repo, here's again the reference to the issue which ultimately triggered the decision of the ATT team to revoke commit rights for krader/siteshwar: segfault with extended globs att/ast#1464. so the truth of it is, that at a certain point, I personally decided to make the ATT team aware and to mince no words what I believed (and still believe) to be the correct decision. that they acted as promptly as they did, a) surprised me and b) shows how serious they took it (correctly, in my. view). you also seem to miss the fact that @gordonwoodhull very politely and diplomatically proposed that ksh2020 just go on in a fork, which would have been perfectly possible. that they (krader and siteshwar) dropped the project like a hot potato the moment they could no longer pretend that their work is the "official" ATT-sanctioned future of ksh is very telling in my view.

EDIT ON REREADING
2. I forgot to address your use of the very strong word "disgraceful" for describing the action of the ATT team. I believe this is really unfair. @gordonwoodhull in my view did an admirable job in providing them an "honourable" way out of the ATT repo and proposing they continue their work in a fork. you somehow seem to ignore that ksh2020 really was on a crassly divergent course from what the AST repo wants to represent. revoking commit rights to that repo (nothing more) sure was an overdue move.

never have been able to do what I am doing now. So I am grateful to them, and I wish the situation could have been fixed without killing their appetite to work on ksh forever.

nobody has "killed" anything. the decision to simply abandon ksh2020 instead of continuing the work in a separate project was entirely theirs.

see above. I only can suggest that you reread through the att/ast issue history what has and what has not been said or proposed by different people at the time. I maintain ksh2020 was heading in a very wrong direction and that mostly, nearly exclusively, was due to krader's approach. that does say nothing about his C capabilities and nothing against salvaging sensible fixes from the ksh2020 code.

So perhaps you can see why I am a little reluctant to join your organisation. It's not only that I

given your assessment, I can. but I don't agree with the assessment :).

don't have time for the slowness, politics, interpersonal stuff, etc. that are inevitable when

in fact, it very well could be that your aversions in this regard are not very different from mine. so far and in my view this is mostly a non-issue in the context of ksh-community.

founding and running an organisation; it's also that you've already shown how you deal with
people you disagree with.

sorry, this is a real non-sequitur to me (to put it mildly). I really believe the different conversations/discussions in the ATT/AST issue history tell a different story. it is now beating a dead horse anyway, but if there was a guy exhibiting agressive and intolerant behaviour there it was krader and krader alone. but whatever...

Having said that, I wish you well, and I hope you'll have great success. I
would be very happy to collaborate on a more informal basis, and I remain absolutely willing to contribute all my work back whenever you're ready to accept it.

amen. :).

[McDutchie]

Thank you for your reply. Your point that the ksh2020 maintainers could have continued in their own fork, just as you are intending to do here, is well taken. I accept that "disgraceful" was too strong a qualification on my part, so I will edit my reply. At the same time I can fully understand why they didn't accept being demoted by AT&T, so they would have to work harder at gaining visibility, etc. For us it's different as we never had AT&T blessing to begin with.

As for "organisation vs. no organisation" being a non-issue, I will just say that there is one repo with significant current activity, and this one ain't it… ;-)

I don't agree that krader1961 was the only person exhibiting aggressive and intolerant behaviour; I distinctly remember being put off by some of his opponents' behaviour as well. But, as you correctly point out, it's a dead horse so I'm happy to leave it and go back to fixing bugs. I just wanted to make my current position clear so we avoid future surprises/drama.