Dear 44Net community,
I think it’s safe to say that the past few months on this mailing list have been, well, a lot. ARDC launched a new portal, which, we learned post-launch, definitely had some major issues. Much of the frustration around this rollout is more than reasonable.
Unfortunately, communications from a very vocal minority crossed a threshold from expressing warranted and reasonable frustration and shifted to hostility, flaming, and trolling. Harassment of some of our employees occurred off-list.
It was at this point that ARDC’s Code of Conduct committee was brought into the discussion. The Committee’s purpose is to confirm whether, in fact, a dispute is simply a disagreement, or whether it is harassment, trolling, or similar. Necessarily, the Committee must not include people directly impacted by the dispute, so that they can, to the best of their ability, respond impartially. At an even higher level, their job is to uphold ARDC’s values – notably of respect, accountability, and fairness.
https://www.ardc.net/about/values/
In the case of this dispute – as a previous mail forwarded to these lists described – the committee’s decision was to temporarily suspend the accounts of one or more mailing list members.
Some mailing list members may not like this decision. Other mailing list members have applauded it. Some mailing list members are so frustrated by the entire situation that they are unsubscribing and/or relinquishing their posts (which, to me, is disheartening but understandable).
In our decisions, we are not going to make everyone happy. It is the nature of making decisions in an environment with a variety of opinions. To maintain room for diversity of thought, our commitment must be, first and foremost, to keeping healthy, supportive spaces for dialogue, discussion, and learning. This means, at times, (ideally temporary) removal of people whose behaviors significantly detract from those goals.
In the breathing room that has been created from the temporary suspension, I hope that we may all get back to solving technical and policy problems, and to engaging in open and respectful dialogue.
Thank you for your attention.
73, Rosy
Never easy, especially when emotions are running high. Thanks to the ARDC team and the Code of Conduct committee, things on the list certainly were getting a bit much. I will be glad to see it get back to previous discussions.
-Colin
-----Original Message----- From: Rosy Schechter - KJ7RYV via 44net 44net@mailman.ampr.org Sent: Tuesday, July 2, 2024 10:39 To: Amprnet 44 Net 44net@mailman.ampr.org; 44net@ardc.groups.io Subject: [44net] a communication about communications
Dear 44Net community,
I think it’s safe to say that the past few months on this mailing list have been, well, a lot. ARDC launched a new portal, which, we learned post-launch, definitely had some major issues. Much of the frustration around this rollout is more than reasonable.
Unfortunately, communications from a very vocal minority crossed a threshold from expressing warranted and reasonable frustration and shifted to hostility, flaming, and trolling. Harassment of some of our employees occurred off-list.
It was at this point that ARDC’s Code of Conduct committee was brought into the discussion. The Committee’s purpose is to confirm whether, in fact, a dispute is simply a disagreement, or whether it is harassment, trolling, or similar. Necessarily, the Committee must not include people directly impacted by the dispute, so that they can, to the best of their ability, respond impartially. At an even higher level, their job is to uphold ARDC’s values – notably of respect, accountability, and fairness.
https://www.ardc.net/about/values/
In the case of this dispute – as a previous mail forwarded to these lists described – the committee’s decision was to temporarily suspend the accounts of one or more mailing list members.
Some mailing list members may not like this decision. Other mailing list members have applauded it. Some mailing list members are so frustrated by the entire situation that they are unsubscribing and/or relinquishing their posts (which, to me, is disheartening but understandable).
In our decisions, we are not going to make everyone happy. It is the nature of making decisions in an environment with a variety of opinions. To maintain room for diversity of thought, our commitment must be, first and foremost, to keeping healthy, supportive spaces for dialogue, discussion, and learning. This means, at times, (ideally temporary) removal of people whose behaviors significantly detract from those goals.
In the breathing room that has been created from the temporary suspension, I hope that we may all get back to solving technical and policy problems, and to engaging in open and respectful dialogue.
Thank you for your attention.
73, Rosy
-- Rosy Schechter - KJ7RYV Executive Director Amateur Radio Digital Communications (ARDC) ardc.net _______________________________________________ 44net mailing list -- 44net@mailman.ampr.org To unsubscribe send an email to 44net-leave@mailman.ampr.org
On 02/07/2024 17:38, Rosy Schechter - KJ7RYV via 44net wrote:
It was at this point that ARDC’s Code of Conduct committee was brought into the discussion. The Committee’s purpose is to confirm whether, in fact, a dispute is simply a disagreement, or whether it is harassment,
Actually, harassment is a crime
In the US, the UK, Australia, Ireland and many other countries, it is the role of a jury or a magistrate to "confirm whether, in fact" somebody is guilty of a crime.
Any other group established arbitrarily to "confirm" a crime is subverting due process.
When any other group established arbitrarily begins making declarations about guilt or innocence in relation to crime they are rolling back everything that we have gained since the Magna Carta in 1215.
Such determinations themselves can then be thought of as harassment.
In 2018, my Outreachy intern wrote a blog about Google monitoring her movements. Google is a sponsor of Outreachy. The Google employees felt she was "harassing" them when all she was doing is saying "NO" to the world's biggest corporate pervert.
https://danielpocock.com/the-harassment-claims-in-full/
Hello Rosy and the team,
thank you to all your team for your investment, especially Chris, who is patient with me.
🙂
Best regards, Ludovic - F5PBG
On Tue, Jul 2, 2024 at 12:38 PM Rosy Schechter via groups.io rosy=ardc.net@groups.io wrote:
Dear 44Net community,
I think it’s safe to say that the past few months on this mailing list have been, well, a lot. ARDC launched a new portal, which, we learned post-launch, definitely had some major issues. Much of the frustration around this rollout is more than reasonable.
Unfortunately, communications from a very vocal minority crossed a threshold from expressing warranted and reasonable frustration and shifted to hostility, flaming, and trolling. Harassment of some of our employees occurred off-list.
It was at this point that ARDC’s Code of Conduct committee was brought into the discussion. The Committee’s purpose is to confirm whether, in fact, a dispute is simply a disagreement, or whether it is harassment, trolling, or similar. Necessarily, the Committee must not include people directly impacted by the dispute, so that they can, to the best of their ability, respond impartially. At an even higher level, their job is to uphold ARDC’s values – notably of respect, accountability, and fairness.
https://www.ardc.net/about/values/
In the case of this dispute – as a previous mail forwarded to these lists described – the committee’s decision was to temporarily suspend the accounts of one or more mailing list members.
Some mailing list members may not like this decision. Other mailing list members have applauded it. Some mailing list members are so frustrated by the entire situation that they are unsubscribing and/or relinquishing their posts (which, to me, is disheartening but understandable).
In our decisions, we are not going to make everyone happy. It is the nature of making decisions in an environment with a variety of opinions. To maintain room for diversity of thought, our commitment must be, first and foremost, to keeping healthy, supportive spaces for dialogue, discussion, and learning. This means, at times, (ideally temporary) removal of people whose behaviors significantly detract from those goals.
In the breathing room that has been created from the temporary suspension, I hope that we may all get back to solving technical and policy problems, and to engaging in open and respectful dialogue.
Thank you for your attention.
Might one offer a different viewpoint for consideration?
Everything happens in some sort of context. In this case, the context is 44Net, how it works, and how that has changed given events in the last 5 or so years, particularly around communication.
I think it's fair to say that administration under Brian Kantor was both collaborative, and what one might describe as, "light touch." Volunteer voices were valued, and administration was spread informally around a group of people; those who were interested in doing the work volunteered and did the work, and were welcomed in doing so. This system was not always perfect (for instance, some coordinators were inattentive, and one former coordinator had a criminal record that made me, at least, extremely uncomfortable whenever I had to interact with him [note: this is not trolling; this was a matter of public record that legitimately freaked me out]).
In the old manner of doing things, the volunteer coordinators expended significant time, energy and in some cases money, to assist with the administration of the network. They developed relationships with users, and bore the brunt of handling day-to-day requests, etc. WB6CYT was the overall anchor holding this crew together, true, but much of the work was outsourced.
After the sale of the /10 (FTR, a move I fully supported and continue to support), this seemed to change. ARDC became much more involved in the daily operation of the network. With the new portal, the role of the coordinators seems greatly reduced. Public requests for public technical discussion involving ARDC-administered software (like AMPRGW) largely goes unanswered, or given perfunctory responses to file a ticket, often with little follow-up. Frankly, it's hard not to feel ignored.
One of the more vocal folks here has been a coordinator. Putting myself in the position of the coordinators, a way to look at this is that the work they had put in: building rapport with users, doing tech support, walking new folks through getting online, was simply discarded. Moreover, sincere offers to help with reducing the backlog of tickets have seemingly been ignored; recall that these folks have already spent considerable amounts of their own time and energy building relationships that could make this easy, and up until the introduction of the new portal, they were entrusted to do just that...why not take them up on something they've already shown aptitude and desire to do? Since we're talking about respectful communication, I can say that, if I were in that situation, I'd probably feel pretty disrespected: wouldn't you?
None of this is to suggest that harassment or outright trolling is useful, or should be excused, let alone tolerated; I'm not a party to the off-list harassment that took place, so I cannot comment on that at all. Nor am I a coordinator and in fairness, I have no insight into what's going on other than what's been publicly shared on this list. But I do have my own experiences and observations to draw on, and taking a step back, what I observe has happened is that the context in which 44net operates has fundamentally changed, but that has not been clearly or sufficiently communicated to the existing stakeholders (and the coordinators and users _are_ stakeholders!).
From my perspective, things have become less collaborative, less experimental, and frankly far less transparent; there seems to be more top-down administration, and a lot less room for volunteer contribution. Pointing out errors in documentation is all well and good, but ignores the considerable areas in which others might usefully contribute.
It's fine for the context to change, of course, but if we want a healthy, vibrant community going forward, it is not ok for that to happen without open communication. And right now, communication from ARDC to the larger community does not feel very open.
As a concrete example, that the rollout of the new portal has been rough has been acknowledged by all parties. But have there been any formal (or even informal!) "lessons learned" recorded in the wake of that? Will they be shared publicly? Is there anything actionable to come out of that? If ARDC were to embark on a similar software project tomorrow, what would be done differently? Would members of the community be invited to be more involved from the start? Would the resulting source code be open? Speaking as a professional software engineer, I can't help but feel if that there had been more engagement from the outset, many of the rougher edges in the portal launch could have been avoided, and --- I admit I'm purely speculating here --- perhaps the present sad situation would never have materialized as a result.
Again, I'm not trying to defend harassment. But the road of respect goes both ways, and I respectfully wonder if ARDC has critically examined its own role here. In my opinion, it could start with more transparency and more communication and engagement with the community at large, and especially with the volunteers who have given much of themselves to the community and asked for little in return. Am I being unreasonable?
- Dan C. (KZ2X)
Well said Dan, it describes very much my own observations.
Bob (Boudewijn) VE3TOK
On 2024-07-02 17:48, Dan Cross via 44net wrote:
On Tue, Jul 2, 2024 at 12:38 PM Rosy Schechter via groups.io rosy=ardc.net@groups.io wrote:
Dear 44Net community,
I think it’s safe to say that the past few months on this mailing list have been, well, a lot. ARDC launched a new portal, which, we learned post-launch, definitely had some major issues. Much of the frustration around this rollout is more than reasonable.
Unfortunately, communications from a very vocal minority crossed a threshold from expressing warranted and reasonable frustration and shifted to hostility, flaming, and trolling. Harassment of some of our employees occurred off-list.
It was at this point that ARDC’s Code of Conduct committee was brought into the discussion. The Committee’s purpose is to confirm whether, in fact, a dispute is simply a disagreement, or whether it is harassment, trolling, or similar. Necessarily, the Committee must not include people directly impacted by the dispute, so that they can, to the best of their ability, respond impartially. At an even higher level, their job is to uphold ARDC’s values – notably of respect, accountability, and fairness.
https://www.ardc.net/about/values/
In the case of this dispute – as a previous mail forwarded to these lists described – the committee’s decision was to temporarily suspend the accounts of one or more mailing list members.
Some mailing list members may not like this decision. Other mailing list members have applauded it. Some mailing list members are so frustrated by the entire situation that they are unsubscribing and/or relinquishing their posts (which, to me, is disheartening but understandable).
In our decisions, we are not going to make everyone happy. It is the nature of making decisions in an environment with a variety of opinions. To maintain room for diversity of thought, our commitment must be, first and foremost, to keeping healthy, supportive spaces for dialogue, discussion, and learning. This means, at times, (ideally temporary) removal of people whose behaviors significantly detract from those goals.
In the breathing room that has been created from the temporary suspension, I hope that we may all get back to solving technical and policy problems, and to engaging in open and respectful dialogue.
Thank you for your attention.
Might one offer a different viewpoint for consideration?
Everything happens in some sort of context. In this case, the context is 44Net, how it works, and how that has changed given events in the last 5 or so years, particularly around communication.
I think it's fair to say that administration under Brian Kantor was both collaborative, and what one might describe as, "light touch." Volunteer voices were valued, and administration was spread informally around a group of people; those who were interested in doing the work volunteered and did the work, and were welcomed in doing so. This system was not always perfect (for instance, some coordinators were inattentive, and one former coordinator had a criminal record that made me, at least, extremely uncomfortable whenever I had to interact with him [note: this is not trolling; this was a matter of public record that legitimately freaked me out]).
In the old manner of doing things, the volunteer coordinators expended significant time, energy and in some cases money, to assist with the administration of the network. They developed relationships with users, and bore the brunt of handling day-to-day requests, etc. WB6CYT was the overall anchor holding this crew together, true, but much of the work was outsourced.
After the sale of the /10 (FTR, a move I fully supported and continue to support), this seemed to change. ARDC became much more involved in the daily operation of the network. With the new portal, the role of the coordinators seems greatly reduced. Public requests for public technical discussion involving ARDC-administered software (like AMPRGW) largely goes unanswered, or given perfunctory responses to file a ticket, often with little follow-up. Frankly, it's hard not to feel ignored.
One of the more vocal folks here has been a coordinator. Putting myself in the position of the coordinators, a way to look at this is that the work they had put in: building rapport with users, doing tech support, walking new folks through getting online, was simply discarded. Moreover, sincere offers to help with reducing the backlog of tickets have seemingly been ignored; recall that these folks have already spent considerable amounts of their own time and energy building relationships that could make this easy, and up until the introduction of the new portal, they were entrusted to do just that...why not take them up on something they've already shown aptitude and desire to do? Since we're talking about respectful communication, I can say that, if I were in that situation, I'd probably feel pretty disrespected: wouldn't you?
None of this is to suggest that harassment or outright trolling is useful, or should be excused, let alone tolerated; I'm not a party to the off-list harassment that took place, so I cannot comment on that at all. Nor am I a coordinator and in fairness, I have no insight into what's going on other than what's been publicly shared on this list. But I do have my own experiences and observations to draw on, and taking a step back, what I observe has happened is that the context in which 44net operates has fundamentally changed, but that has not been clearly or sufficiently communicated to the existing stakeholders (and the coordinators and users _are_ stakeholders!).
From my perspective, things have become less collaborative, less experimental, and frankly far less transparent; there seems to be more top-down administration, and a lot less room for volunteer contribution. Pointing out errors in documentation is all well and good, but ignores the considerable areas in which others might usefully contribute.
It's fine for the context to change, of course, but if we want a healthy, vibrant community going forward, it is not ok for that to happen without open communication. And right now, communication from ARDC to the larger community does not feel very open.
As a concrete example, that the rollout of the new portal has been rough has been acknowledged by all parties. But have there been any formal (or even informal!) "lessons learned" recorded in the wake of that? Will they be shared publicly? Is there anything actionable to come out of that? If ARDC were to embark on a similar software project tomorrow, what would be done differently? Would members of the community be invited to be more involved from the start? Would the resulting source code be open? Speaking as a professional software engineer, I can't help but feel if that there had been more engagement from the outset, many of the rougher edges in the portal launch could have been avoided, and --- I admit I'm purely speculating here --- perhaps the present sad situation would never have materialized as a result.
Again, I'm not trying to defend harassment. But the road of respect goes both ways, and I respectfully wonder if ARDC has critically examined its own role here. In my opinion, it could start with more transparency and more communication and engagement with the community at large, and especially with the volunteers who have given much of themselves to the community and asked for little in return. Am I being unreasonable?
- Dan C. (KZ2X)
44net mailing list -- 44net@mailman.ampr.org To unsubscribe send an email to 44net-leave@mailman.ampr.org
Mine too! I have been a coordinator for nearly 30 years, both in the times of the classic packet radio network and the current "hamnet", and I have done a lot of work on guiding the users and building the network. Out network is very active, we have lots of users and systems. The move to the new portal has been made with the assumption that all of the existing allocations were abandoned and unused, and we can start fresh. But over here that isn't the case at all! The new method of administration has its pros and cons, but the lack of a migration plan has caused a lot of disruption here. It simply isn't practical to have >500 amateurs of different skills migrate from a centrally administered system to creating their own account, claiming their own addresses and allocations, and managing them, and have the system effectively out of operation during the transition. That combined with the fact that each and every request I made via the portal ticket system to have the major issues solved was turned down has made me sad and disappointed in ARDC. I got the impression that for everything I carefully thought out to resolve the situation, a new rule was invented on the spot as to why that could not be granted. I have tried to keep silent and just ignore the whole thing, hoping that it would finally be resolved, but I can fully understand why people become frustrated over this. As I wrote several times in tickets: why can't we all just cooperate in a nice hobby where everyone has fun and everyone takes the aspects of the hobby that they like, as it should be in ham radio? Why should there be all those procedures, rules, administration, and denial of requests made by other people who are trying to have fun? I do not understand it.
Rob PE1CHL
On 2024-07-03 02:15, Boudewijn (Bob) Tenty via 44net wrote:
Well said Dan, it describes very much my own observations.
Bob (Boudewijn) VE3TOK
Hi all,
Responding briefly because today is a holiday in the U.S., but I don’t want to leave this conversation about transparency unaddressed until next week. I’m sharing some thoughts here on the subject of transparency, as well as next steps for how we might address some of the concerns shared in this thread and beyond.
Sincere thank you Dan and others for voicing your thoughts and opinions here. These points in particular I take to heart:
<quote>
After the sale of the /10 (FTR, a move I fully supported and continue to support), this seemed to change. ARDC became much more involved in the daily operation of the network. With the new portal, the role of the coordinators seems greatly reduced. Public requests for public technical discussion involving ARDC-administered software (like AMPRGW) largely goes unanswered, or given perfunctory responses to file a ticket, often with little follow-up. Frankly, it's hard not to feel ignored.
/
From my perspective, things have become less collaborative, less experimental, and frankly far less transparent; there seems to be more top-down administration, and a lot less room for volunteer contribution. Pointing out errors in documentation is all well and good, but ignores the considerable areas in which others might usefully contribute.
</quote>
I agree that we could do a better job with being transparent. A clear learning from the most recent launch was that involving some of you in testing the portal prior to launch would have been helpful, as well as opening up discussions around any potential shifts in policies. There are certainly more lessons learned, and in the coming weeks - following a request from some of y'all – I’ll post a more comprehensive list. (Some staff members are in and out of town for the next couple of weeks, and I want to make sure to review with everyone prior to sharing, so thanks for your patience there.)
On a related note, please understand that any lack of communication or greater engagement on our part has more to do with capacity (or lack thereof) than anything else. One of the key functions of our upcoming Technical Department Manager hire will be to interface with this community on a more regular basis.
Even with such a hire, though, one thing is also true that is worth mentioning: there is no possible way to replace Brian Kantor. Inevitably, when someone exits an organization, something shifts. When someone enters, there is also a shift. And when Brian and ARDC’s board decided to sell part of the address space, there was a major shift there too. From the perspective of those of you who have been around for 10 years or more, I can see how these shifts have felt quite stark.
Additionally, Brian used to run everything. When he passed away suddenly, he left no playbook.
Given that, I am interested in learning from you how he engaged with you so that we can do a better job, to the best of our ability. Clearly newsletters and sharing announcements with the list isn’t cutting it. What specific and actionable steps can we take to better address your concerns?
In addition to getting your feedback here, we’ll be talking about this a bit at the next Regional Coordinators’ meeting (July 27). We aim to hold these meetings regularly (at least every 1-3 months, depending on everyone's availability).
Ok, this is a much longer email than I intended, but it’s as they say - if I had more time, I would write a shorter letter. Nevertheless, the point I’m trying to get across – which I hope is received – is that we are willing to learn from our mistakes and to do things better. We are all, in fact, on the same team – growing pains and all.
For those in the US, I wish you a happy Independence Day. For everyone else, I hope you have a good weekend, and I look forward to picking up the conversation next week.
73, Rosy
Rosy Schechter - KJ7RYV Executive Director Amateur Radio Digital Communications (ARDC) ardc.net
On Thu, Jul 4, 2024 at 3:45 PM Rosy Schechter via groups.io rosy=ardc.net@groups.io wrote:
Responding briefly because today is a holiday in the U.S., but I don’t want to leave this conversation about transparency unaddressed until next week. I’m sharing some thoughts here on the subject of transparency, as well as next steps for how we might address some of the concerns shared in this thread and beyond.
Sincere thank you Dan and others for voicing your thoughts and opinions here. These points in particular I take to heart:
Happy to do it. As you noted, it was a holiday, so please excuse the delayed response.
<quote> > After the sale of the /10 (FTR, a move I fully supported and continue > to support), this seemed to change. ARDC became much more involved in > the daily operation of the network. With the new portal, the role of > the coordinators seems greatly reduced. Public requests for public > technical discussion involving ARDC-administered software (like > AMPRGW) largely goes unanswered, or given perfunctory responses to > file a ticket, often with little follow-up. Frankly, it's hard not to > feel ignored. > > From my perspective, things have become less collaborative, less > experimental, and frankly far less transparent; there seems to be more > top-down administration, and a lot less room for volunteer > contribution. Pointing out errors in documentation is all well and > good, but ignores the considerable areas in which others might > usefully contribute.
I agree that we could do a better job with being transparent. A clear learning from the most recent launch was that involving some of you in testing the portal prior to launch would have been helpful, as well as opening up discussions around any potential shifts in policies. There are certainly more lessons learned, and in the coming weeks - following a request from some of y'all – I’ll post a more comprehensive list. (Some staff members are in and out of town for the next couple of weeks, and I want to make sure to review with everyone prior to sharing, so thanks for your patience there.)
On a related note, please understand that any lack of communication or greater engagement on our part has more to do with capacity (or lack thereof) than anything else. One of the key functions of our upcoming Technical Department Manager hire will be to interface with this community on a more regular basis.
That's fair.
Even with such a hire, though, one thing is also true that is worth mentioning: there is no possible way to replace Brian Kantor. Inevitably, when someone exits an organization, something shifts. When someone enters, there is also a shift. And when Brian and ARDC’s board decided to sell part of the address space, there was a major shift there too. From the perspective of those of you who have been around for 10 years or more, I can see how these shifts have felt quite stark.
Additionally, Brian used to run everything. When he passed away suddenly, he left no playbook.
Given that, I am interested in learning from you how he engaged with you so that we can do a better job, to the best of our ability. Clearly newsletters and sharing announcements with the list isn’t cutting it. What specific and actionable steps can we take to better address your concerns?
As you said, there's no replacing Brian. Things change and the context shifts; in that spirit, I wonder if the most useful questions are less about how he did things and more about how to build the community we'd all like going forward?
It's difficult to make concrete suggestions from my perspective: I'm not a coordinator, just one random person. Moreover, the portal probably isn't going to support some of these things, and will require development effort to implement properly. Regardless, if someone put the question to me, here are a few things I'd throw out there:
1. Let the coordinators take on ownership of tickets for users and allocations that fall under their purview. Let them take care of validating those users for whom they have prior relationships, for example.
2. Provide some sort of "batch mode" capability for coordinators to make updates on behalf of users. Consider DNS for example: I like the new system of being able to modify my own records, but I've been a (near) daily user of the Internet for other 30 years and have been running machines on the network for nearly as long. Some folks aren't going to be comfortable making updates and are going to lean on their coordinators to help guide them through the process. The current process, while nice, is also rather labor intensive, in that each record has to be added manually through the user interface.
3. Provide some sort of programmatic API for the portal. A batch interface would be a strict improvement over what's there now; I imagine that it would be even nicer if someone could just run a script.
4. Have technical discussions out in the open. The ticket system is all well-and-good, but I pointed out an issue with reverse DNS queries being (presumably) eaten by AMPRGW back in early May. I put some effort into writing up a summary, complete with my experimental method and observations. I was asked to file a ticket; I did, and tried to link to the discussion on the list for context; I was asked to put that data into a ticket, but the ticket system really isn't built for that: the textual input fields lack the space and formatting abilities to incorporate what I'd already written. I ended up putting it all into a text file and attacking it to the ticket, but it's not clear to me that that was ever read. It ended up being an enormously frustrating experience, and while I'm quite certain it was not intentional, frankly it all felt extremely disrespectful of the time I put into the matter. Similarly, I tried to figure out why AMPRGW stopped passing traffic when a router reboots; my investigations suggested that this is due to AMPRGW seeing ICMP unreachable messages for the endpoint during the reboot blackout window. I asked publically for confirmation of this _so that I could update the wiki with that information_; after a few days, I was again asked to file a ticket. Eventually I kinda-sorta got confirmation, but it was painful. The bottom line is that an ersatz ticketing system is not a substitute for a public forum for open discussion of technical matters.
5. In the spirit of openness, release the portal software as open source, putting it on the ARDC git server. Since much of the above hinges on (presumably) enhancements to the portal, it seems like it might be helpful to leverage the community in making those enhancements, which can't reasonably be done _unless_ the software is open. Indeed, I can't think of a good reason not to do this; to forestall what I suspect would be an objection, security is not a good reason: "security through obscurity" is a well-known anti-pattern in the software world and rarely works. It's much better when software is open for inspection and improvement by all.
6. Much of the current state of affairs comes from the rocky rollout of the portal. There's little we can do about this now, but a publically accessible post-mortem (preferably following the "blameless" format) about the launch: what went well, what went poorly, lessons learned and what could be done in the future to mitigate pain points, would be very well received, I imagine.
7. I know that plans are underway to deprecate the tunnel infrastructure and replace it with a set of VPNs that are independently peered with the Internet at large (at least, as I understand it). I think it would be helpful to publicly commit to a transition plan for this, including a public test period, a defined set of go/no-go criteria, and a plan for a rollback if it doesn't go well. In general, commit to involving the community early for any large-scale infrastructure projects ARDC takes on in the future.
Anyway, these are the things I sort of thought of off the top of my head. In fairness, I recognize some are easier than others, and I hope other folks will chime in with thoughts as well.
- Dan C. (KZ2X)
In addition to getting your feedback here, we’ll be talking about this a bit at the next Regional Coordinators’ meeting (July 27). We aim to hold these meetings regularly (at least every 1-3 months, depending on everyone's availability).
Ok, this is a much longer email than I intended, but it’s as they say - if I had more time, I would write a shorter letter. Nevertheless, the point I’m trying to get across – which I hope is received – is that we are willing to learn from our mistakes and to do things better. We are all, in fact, on the same team – growing pains and all.
For those in the US, I wish you a happy Independence Day. For everyone else, I hope you have a good weekend, and I look forward to picking up the conversation next week.
73, Rosy
Rosy Schechter - KJ7RYV Executive Director Amateur Radio Digital Communications (ARDC) ardc.net
-=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#320): https://ardc.groups.io/g/44net/message/320 Mute This Topic: https://groups.io/mt/107043561/481017 Group Owner: 44net+owner@ardc.groups.io Unsubscribe: https://ardc.groups.io/g/44net/leave/13272901/481017/1172239927/xyzzy [crossd@gmail.com] -=-=-=-=-=-=-=-=-=-=-=-
Hi Dan,
Just to correct a misconception:
On 8 Jul 2024, at 20:45, Dan Cross via 44net 44net@mailman.ampr.org wrote: : 7. I know that plans are underway to deprecate the tunnel infrastructure and replace it with a set of VPNs that are independently peered with the Internet at large (at least, as I understand it). I think it would be helpful to publicly commit to a transition plan for this, including a public test period, a defined set of go/no-go criteria, and a plan for a rollback if it doesn't go well. In general, commit to involving the community early for any large-scale infrastructure projects ARDC takes on in the future.
Plans are not underway to deprecate the IPIP mesh tunnel infrastructure.
Yes, ARDC are developing POPs that utilise VPN technology, and if all goes well I am sure that some users of the IPIP tunnels will likely migrate to the POPs, also newcomers to 44Net will find the POPs a much easier on-ramp than the IPIP system, so IPIP use may well reduce over time. To re-iterate ARDC have no plans to deprecate the IPIP infrastructure, ARDC will continue to support its use for as long as the membership is using it.
73, Chris - G1FEF
On Mon, Jul 8, 2024 at 4:21 PM Chris chris@ardc.net wrote:
Hi Dan,
Just to correct a misconception:
On 8 Jul 2024, at 20:45, Dan Cross via 44net 44net@mailman.ampr.org wrote: : 7. I know that plans are underway to deprecate the tunnel infrastructure and replace it with a set of VPNs that are independently peered with the Internet at large (at least, as I understand it). I think it would be helpful to publicly commit to a transition plan for this, including a public test period, a defined set of go/no-go criteria, and a plan for a rollback if it doesn't go well. In general, commit to involving the community early for any large-scale infrastructure projects ARDC takes on in the future.
Plans are not underway to deprecate the IPIP mesh tunnel infrastructure.
Yes, ARDC are developing POPs that utilise VPN technology, and if all goes well I am sure that some users of the IPIP tunnels will likely migrate to the POPs, also newcomers to 44Net will find the POPs a much easier on-ramp than the IPIP system, so IPIP use may well reduce over time. To re-iterate ARDC have no plans to deprecate the IPIP infrastructure, ARDC will continue to support its use for as long as the membership is using it.
Thanks for the correction, Chris, and apologies for spreading misinformation.
I was under the understanding, however, that the intent is to turn down AMPRGW at some point; would tunnel operators point at one of the other POPs for e.g. a default route, then? Thanks.
- Dan C.
On 8 Jul 2024, at 21:28, Dan Cross crossd@gmail.com wrote:
On Mon, Jul 8, 2024 at 4:21 PM Chris chris@ardc.net wrote:
Hi Dan,
Just to correct a misconception:
On 8 Jul 2024, at 20:45, Dan Cross via 44net 44net@mailman.ampr.org wrote: : 7. I know that plans are underway to deprecate the tunnel infrastructure and replace it with a set of VPNs that are independently peered with the Internet at large (at least, as I understand it). I think it would be helpful to publicly commit to a transition plan for this, including a public test period, a defined set of go/no-go criteria, and a plan for a rollback if it doesn't go well. In general, commit to involving the community early for any large-scale infrastructure projects ARDC takes on in the future.
Plans are not underway to deprecate the IPIP mesh tunnel infrastructure.
Yes, ARDC are developing POPs that utilise VPN technology, and if all goes well I am sure that some users of the IPIP tunnels will likely migrate to the POPs, also newcomers to 44Net will find the POPs a much easier on-ramp than the IPIP system, so IPIP use may well reduce over time. To re-iterate ARDC have no plans to deprecate the IPIP infrastructure, ARDC will continue to support its use for as long as the membership is using it.
Thanks for the correction, Chris, and apologies for spreading misinformation.
I was under the understanding, however, that the intent is to turn down AMPRGW at some point;
Not that I am aware of.
Chris
would tunnel operators point at one of the other POPs for e.g. a default route, then? Thanks.
On Mon, Jul 8, 2024 at 4:34 PM Chris chris@ardc.net wrote:
On 8 Jul 2024, at 21:28, Dan Cross crossd@gmail.com wrote: On Mon, Jul 8, 2024 at 4:21 PM Chris chris@ardc.net wrote:
Hi Dan,
Just to correct a misconception:
On 8 Jul 2024, at 20:45, Dan Cross via 44net 44net@mailman.ampr.org wrote: : 7. I know that plans are underway to deprecate the tunnel infrastructure and replace it with a set of VPNs that are independently peered with the Internet at large (at least, as I understand it). I think it would be helpful to publicly commit to a transition plan for this, including a public test period, a defined set of go/no-go criteria, and a plan for a rollback if it doesn't go well. In general, commit to involving the community early for any large-scale infrastructure projects ARDC takes on in the future.
Plans are not underway to deprecate the IPIP mesh tunnel infrastructure.
Yes, ARDC are developing POPs that utilise VPN technology, and if all goes well I am sure that some users of the IPIP tunnels will likely migrate to the POPs, also newcomers to 44Net will find the POPs a much easier on-ramp than the IPIP system, so IPIP use may well reduce over time. To re-iterate ARDC have no plans to deprecate the IPIP infrastructure, ARDC will continue to support its use for as long as the membership is using it.
Thanks for the correction, Chris, and apologies for spreading misinformation.
I was under the understanding, however, that the intent is to turn down AMPRGW at some point;
Not that I am aware of.
In ticket 2787 you told me that the gateway _server_ is being deprecated; I imagine I misinterpreted that to mean the gateway _service_. However, in light of what you wrote above, I can interpret this one of two other ways: a) the physical server is being deprecated and replaced, or b) the existing AMPRGW software is being replaced (or there are plans to replace it). It would be good to have a solid understanding of what the actual plan is.
This is an illustrative example of why, IMHO, it's better to have technical discussions in the open.
- Dan C.
Hi Dan,
In ticket 2787 you told me that the gateway _server_ is being deprecated;
I said we are hoping to deprecate the gateway server and then I went on to explain that it would be replaced, not turned down.
Not only is the server very old, but the code it is running has bugs (as you pointed out). I would like to see it replaced with more modern hardware running updated software that is easier to maintain and update.
However, this is not a priority at the moment, so very unlikely to be worked on in the short to medium term.
73, Chris - G1FEF
I imagine I misinterpreted that to mean the gateway _service_. However, in light of what you wrote above, I can interpret this one of two other ways: a) the physical server is being deprecated and replaced, or b) the existing AMPRGW software is being replaced (or there are plans to replace it). It would be good to have a solid understanding of what the actual plan is.
This is an illustrative example of why, IMHO, it's better to have technical discussions in the open.
- Dan C.
On Mon, Jul 8, 2024 at 4:48 PM Chris chris@ardc.net wrote:
Hi Dan,
In ticket 2787 you told me that the gateway _server_ is being deprecated;
I said we are hoping to deprecate the gateway server and then I went on to explain that it would be replaced, not turned down.
Indeed; as I said I misinterpreted.
But I think this again illustrates why conversations like this are best in the open: language is ambiguous, and _replacing_ a server can mean different things depending on the context. For instance, if one replaces an FTP server with an HTTP server, that's very different from replacing one version of an HTTP server with another version of that same server. These sorts of ambiguities can lead to the kind of "telephone" games one sees in this very thread. On the other hand, when we have these conversations in the open, things tend towards more precision and misinterpretations can be corrected quickly.
Not only is the server very old, but the code it is running has bugs (as you pointed out). I would like to see it replaced with more modern hardware running updated software that is easier to maintain and update.
However, this is not a priority at the moment, so very unlikely to be worked on in the short to medium term.
This sounds like a rather ambitious project! I have high hopes this will be done in the open.
Thanks,
- Dan C.
73, Chris - G1FEF
I imagine I misinterpreted that to mean the gateway _service_. However, in light of what you wrote above, I can interpret this one of two other ways: a) the physical server is being deprecated and replaced, or b) the existing AMPRGW software is being replaced (or there are plans to replace it). It would be good to have a solid understanding of what the actual plan is.
This is an illustrative example of why, IMHO, it's better to have technical discussions in the open.
- Dan C.
On 9/7/24 6:21 am, Chris via 44net wrote:
Plans are not underway to deprecate the IPIP mesh tunnel infrastructure.
Yes, ARDC are developing POPs that utilise VPN technology, and if all goes well I am sure that some users of the IPIP tunnels will likely migrate to the POPs, also newcomers to 44Net will find the POPs a much easier on-ramp than the IPIP system, so IPIP use may well reduce over time. To re-iterate ARDC have no plans to deprecate the IPIP infrastructure, ARDC will continue to support its use for as long as the membership is using it.
That's good to hear, because I'm happy with how my virtual LAN is working with my remote IPIP gateway. I think the VPN POPs are going to be great, but for my setup, I can also see a higher chance of suboptimal routing, compared to IPIP. As technology changes, however, I will review my interconnection.
Hi Dan ++
Thanks for these thoughts; they are super helpful. Responding in line...
As you said, there's no replacing Brian. Things change and the context shifts; in that spirit, I wonder if the most useful questions are less about how he did things and more about how to build the community we'd all like going forward?
I like this framing :)
It's difficult to make concrete suggestions from my perspective: I'm not a coordinator, just one random person. Moreover, the portal probably isn't going to support some of these things, and will require development effort to implement properly. Regardless, if someone put the question to me, here are a few things I'd throw out there:
- Let the coordinators take on ownership of tickets for users and
allocations that fall under their purview. Let them take care of validating those users for whom they have prior relationships, for example.
- Provide some sort of "batch mode" capability for coordinators to
make updates on behalf of users. Consider DNS for example: I like the new system of being able to modify my own records, but I've been a (near) daily user of the Internet for other 30 years and have been running machines on the network for nearly as long. Some folks aren't going to be comfortable making updates and are going to lean on their coordinators to help guide them through the process. The current process, while nice, is also rather labor intensive, in that each record has to be added manually through the user interface.
These are definitely items we want to discuss with coordinators. We both want to give folks such abilities but also ensure that they are being responsive / not only giving allocations to their friends and no one else. (This is something that's been brought to our attention a number of times.) There are also some GDPR privacy concerns around PII that have been brought up; not impossible to deal with but will require some real thought and coordination.
The question I'm thus asking is: how can we make sure that coordinators have agency while also ensuring fairness to users, as well as sensitivity of data, etc.?
- Provide some sort of programmatic API for the portal. A batch
interface would be a strict improvement over what's there now; I imagine that it would be even nicer if someone could just run a script.
As Chris mentioned, this is in the works :)
- Have technical discussions out in the open. The ticket system is
all well-and-good, but I pointed out an issue with reverse DNS queries being (presumably) eaten by AMPRGW back in early May. I put some effort into writing up a summary, complete with my experimental method and observations. I was asked to file a ticket; I did, and tried to link to the discussion on the list for context; I was asked to put that data into a ticket, but the ticket system really isn't built for that: the textual input fields lack the space and formatting abilities to incorporate what I'd already written. I ended up putting it all into a text file and attacking it to the ticket, but it's not clear to me that that was ever read. It ended up being an enormously frustrating experience, and while I'm quite certain it was not intentional, frankly it all felt extremely disrespectful of the time I put into the matter. Similarly, I tried to figure out why AMPRGW stopped passing traffic when a router reboots; my investigations suggested that this is due to AMPRGW seeing ICMP unreachable messages for the endpoint during the reboot blackout window. I asked publically for confirmation of this _so that I could update the wiki with that information_; after a few days, I was again asked to file a ticket. Eventually I kinda-sorta got confirmation, but it was painful. The bottom line is that an ersatz ticketing system is not a substitute for a public forum for open discussion of technical matters.
I definitely see what you are saying here and agree that the tickets are not a replacement for a public forum. I think the trick will be to find the balance between, "This is a bug / admin function that doesn't need everyone's attention" (ticket) vs. "I have a technical question that I'd like support from the community to address" (forum). It's about finding the right pathway for each item, and what you're saying here gives me some ideas for items we can implement.
- In the spirit of openness, release the portal software as open
source, putting it on the ARDC git server. Since much of the above hinges on (presumably) enhancements to the portal, it seems like it might be helpful to leverage the community in making those enhancements, which can't reasonably be done _unless_ the software is open. Indeed, I can't think of a good reason not to do this; to forestall what I suspect would be an objection, security is not a good reason: "security through obscurity" is a well-known anti-pattern in the software world and rarely works. It's much better when software is open for inspection and improvement by all.
Agree. This is part of the plan. We'd like to kick it around a little more internally before doing so, and also to make sure that we are ready to handle feature requests / issues / pull requests etc. I fear that if we don't do that in advance, we'll set ourselves up another less-than-awesome rollout. I...don't think we need another one of those ;)
This item is also towards the top of the list for our forthcoming Technical Department Manager.
- Much of the current state of affairs comes from the rocky rollout
of the portal. There's little we can do about this now, but a publically accessible post-mortem (preferably following the "blameless" format) about the launch: what went well, what went poorly, lessons learned and what could be done in the future to mitigate pain points, would be very well received, I imagine.
Yep.
- I know that plans are underway to deprecate the tunnel
infrastructure and replace it with a set of VPNs that are independently peered with the Internet at large (at least, as I understand it). I think it would be helpful to publicly commit to a transition plan for this, including a public test period, a defined set of go/no-go criteria, and a plan for a rollback if it doesn't go well. In general, commit to involving the community early for any large-scale infrastructure projects ARDC takes on in the future.
Chris already responded about this one, so I'll leave it.
Anyway, these are the things I sort of thought of off the top of my head. In fairness, I recognize some are easier than others, and I hope other folks will chime in with thoughts as well.
Thanks again so much, Dan. I appreciate and agree with much of your feedback, and our aim is to implement much of it. It will take time and some more people (we're working on it!), but I'm confident we'll get there.
73, Rosy
On Mon, Jul 8, 2024 at 5:13 PM Rosy Schechter via groups.io rosy=ardc.net@groups.io wrote:
Hi Dan ++
Thanks for these thoughts; they are super helpful. Responding in line...
Thanks, Rosy.
As you said, there's no replacing Brian. Things change and the context shifts; in that spirit, I wonder if the most useful questions are less about how he did things and more about how to build the community we'd all like going forward?
I like this framing :)
It's difficult to make concrete suggestions from my perspective: I'm not a coordinator, just one random person. Moreover, the portal probably isn't going to support some of these things, and will require development effort to implement properly. Regardless, if someone put the question to me, here are a few things I'd throw out there:
- Let the coordinators take on ownership of tickets for users and
allocations that fall under their purview. Let them take care of validating those users for whom they have prior relationships, for example.
- Provide some sort of "batch mode" capability for coordinators to
make updates on behalf of users. Consider DNS for example: I like the new system of being able to modify my own records, but I've been a (near) daily user of the Internet for other 30 years and have been running machines on the network for nearly as long. Some folks aren't going to be comfortable making updates and are going to lean on their coordinators to help guide them through the process. The current process, while nice, is also rather labor intensive, in that each record has to be added manually through the user interface.
These are definitely items we want to discuss with coordinators. We both want to give folks such abilities but also ensure that they are being responsive / not only giving allocations to their friends and no one else. (This is something that's been brought to our attention a number of times.) There are also some GDPR privacy concerns around PII that have been brought up; not impossible to deal with but will require some real thought and coordination.
The question I'm thus asking is: how can we make sure that coordinators have agency while also ensuring fairness to users, as well as sensitivity of data, etc.?
I know this is an open-ended question to generate discussion, but (and I'm just spitballing here) I'd think one could do it with some means of hierarchy, implemented in the portal. A ticket should be addressable to an individual for handling (e.g., a coordinator) but there ought to be some method for "escalation" if it's determined that the ticket falls outside of the purview of the coordinator (or if they're just unresponsive or otherwise uncool about helping a user out). Perhaps this could be tied to time (7 days with no response and the ticket gets auto-reassigned to someone else) or explicitly set up by the ticket assignee ("I'm going on vacation and will be out of town for a month. Please contact $OTHERPERSON instead...").
I'm sympathetic to the privacy concerns.
- Provide some sort of programmatic API for the portal. A batch
interface would be a strict improvement over what's there now; I imagine that it would be even nicer if someone could just run a script.
As Chris mentioned, this is in the works :)
I think he mentioned a replacement for AMPRGW infrastructure and software is being considered; if he mentioned API-based access to the portal, I'm afraid I missed it.
- Have technical discussions out in the open. The ticket system is
all well-and-good, but I pointed out an issue with reverse DNS queries being (presumably) eaten by AMPRGW back in early May. I put some effort into writing up a summary, complete with my experimental method and observations. I was asked to file a ticket; I did, and tried to link to the discussion on the list for context; I was asked to put that data into a ticket, but the ticket system really isn't built for that: the textual input fields lack the space and formatting abilities to incorporate what I'd already written. I ended up putting it all into a text file and attacking it to the ticket, but it's not clear to me that that was ever read. It ended up being an enormously frustrating experience, and while I'm quite certain it was not intentional, frankly it all felt extremely disrespectful of the time I put into the matter. Similarly, I tried to figure out why AMPRGW stopped passing traffic when a router reboots; my investigations suggested that this is due to AMPRGW seeing ICMP unreachable messages for the endpoint during the reboot blackout window. I asked publically for confirmation of this _so that I could update the wiki with that information_; after a few days, I was again asked to file a ticket. Eventually I kinda-sorta got confirmation, but it was painful. The bottom line is that an ersatz ticketing system is not a substitute for a public forum for open discussion of technical matters.
I definitely see what you are saying here and agree that the tickets are not a replacement for a public forum. I think the trick will be to find the balance between, "This is a bug / admin function that doesn't need everyone's attention" (ticket) vs. "I have a technical question that I'd like support from the community to address" (forum). It's about finding the right pathway for each item, and what you're saying here gives me some ideas for items we can implement.
Absolutely. Another category is discovery around, "how does this work?" This being an amateur radio community, people are naturally curious about _how_ things work; when something behaves oddly, I can see how folks want to have that discussion, but that's qualitatively different than a bug report, or request for support.
- In the spirit of openness, release the portal software as open
source, putting it on the ARDC git server. Since much of the above hinges on (presumably) enhancements to the portal, it seems like it might be helpful to leverage the community in making those enhancements, which can't reasonably be done _unless_ the software is open. Indeed, I can't think of a good reason not to do this; to forestall what I suspect would be an objection, security is not a good reason: "security through obscurity" is a well-known anti-pattern in the software world and rarely works. It's much better when software is open for inspection and improvement by all.
Agree. This is part of the plan. We'd like to kick it around a little more internally before doing so, and also to make sure that we are ready to handle feature requests / issues / pull requests etc. I fear that if we don't do that in advance, we'll set ourselves up another less-than-awesome rollout. I...don't think we need another one of those ;)
This item is also towards the top of the list for our forthcoming Technical Department Manager.
That's wonderful to hear. Thanks!
If it helps, there is a bit of a middle ground where a project can be opened up for inspection, but with a disclaimer that it's not ready for external support or handling PRs, etc. We do this at my job for a number of things (one of our core values is transparency and commitment to open source dovetails with that; we try to open up as much as we can, from our OS implementation to our firmware).
- Much of the current state of affairs comes from the rocky rollout
of the portal. There's little we can do about this now, but a publically accessible post-mortem (preferably following the "blameless" format) about the launch: what went well, what went poorly, lessons learned and what could be done in the future to mitigate pain points, would be very well received, I imagine.
Yep.
- I know that plans are underway to deprecate the tunnel
infrastructure and replace it with a set of VPNs that are independently peered with the Internet at large (at least, as I understand it). I think it would be helpful to publicly commit to a transition plan for this, including a public test period, a defined set of go/no-go criteria, and a plan for a rollback if it doesn't go well. In general, commit to involving the community early for any large-scale infrastructure projects ARDC takes on in the future.
Chris already responded about this one, so I'll leave it.
Anyway, these are the things I sort of thought of off the top of my head. In fairness, I recognize some are easier than others, and I hope other folks will chime in with thoughts as well.
Thanks again so much, Dan. I appreciate and agree with much of your feedback, and our aim is to implement much of it. It will take time and some more people (we're working on it!), but I'm confident we'll get there.
I'm happy to chime in, and glad that you find it helpful. Hopefully others will do similarly.
- Dan C.
Hi Dan,
A quick response before the day gets away from me and I am AFK for a few days:
If it helps, there is a bit of a middle ground where a project can be opened up for inspection, but with a disclaimer that it's not ready for external support or handling PRs, etc. We do this at my job for a number of things (one of our core values is transparency and commitment to open source dovetails with that; we try to open up as much as we can, from our OS implementation to our firmware).
This is a great idea, and the team and I just spoke about it today. More soon :)
73, Rosy
Rosy Schechter - KJ7RYV Executive Director Amateur Radio Digital Communications (ARDC) ardc.net
-
Boudewijn (Bob) Tenty -
Chris -
Chuck Gelm NC8Q -
Colin Bodor -
Dan Cross -
Daniel Pocock -
f5pbg＠free.fr -
Rob PE1CHL -
Rosy Schechter - KJ7RYV -
Tony Langdon