Replies: 43 comments 190 replies
|
previous discussion: https://github.com/dotnet/roslyn/issues/2060 notable things missing:
|
|
extension members |
|
Adding comment from issue #503: Expression Trees should be extended to support the C# 6.0 null propagating operator so that it can be used with Linq, with a translation to allow providers such as Linq to SQL to add support. The discussion on CodePlex doesn't seemed to have moved over to here, and has been sitting around for 3 years - I think it is worth revisiting, and ideally implementing. An ideal use case is an Outer Join in Linq to Objects. |
|
+1 |
|
I'd like to add my vote for this proposal for the sake of the Moq 4.x mocking library, whose API relies heavily on expression tree lambdas. In particular, support for the following two language features in expression tree lambda would be hugely beneficial to Moq. We could then fix the majority of currently reported bugs, and add some long requested features: I'm happy to further explain my reasons for prioritising these two, but don't want to go off on a tangent unbidden. |
|
With the (awesome) addition of nullable reference types in C# 8.0 having null-propagation operation in expression trees goes from "very important" to "you almost can not use Entity Framework without this feature". Looking forward for knowing the design details of how this could be implemented in order to adapt my LINQ provider. |
I wonder if local functions could be translated to SQL CTEs. |
|
Bump. @gafter, I was wondering whether there's any news from the C# language team about this. I understand that it's probably more interesting to design and plan brand-new language features like Is there any chance this can be given a higher priority? Is there anything the community can do to help advance expression trees? |
|
Async/await support is probably the largest undertaking on that list, but I'd love it. It would significantly help projects performing complex code-gen. |
|
@scalablecory - I totally agree this would be nice. That being said, I question the likeihood of getting I think it would be awesome if we could just get the ball rolling, i. e. we could get agreement from the C# language team and/or the Roslyn team to look at the most basic missing features for a start. (I'd be happy to help in any way I reasonably can.) If we manage to get there, perhaps the more complex features will eventually follow, too. |
|
It was discussed briefly on Monday... noting for example that we currently do not have support for /cc @dotnet/ldm It would help if we had a member of the LDM who championed this (even though it isn't technically a language feature). Any takers? |
|
I think there would need to be a language change to support statements in expression trees because expression tree types explicitly states:
emphasis added. For everything else:
Should it be (libraries that walk these often assume specific patterns; the fact that according to the language spec they appear to be able to change between compiler versions is surprising isn't it)? (OT, it looks like there are numerous spaces missing from this document after the word "expression"; are PRs being accepted for these sorts of things yet?) |
|
@bbarry PRs to fix typos in the spec are most welcome. |
|
@gafer Is there any issue where the changes on the expression API are being discussed? |
|
Not that I am aware |
|
I don’t see ?. in expression trees anywhere in the selected features for c# 8. Any chance it will be included? Is considered a part of nullable reference types? I had three bugs in production because of SQL implicit nulls in left joins... What can we do to help this move forward? |
|
A lot of the concern on this thread speaks to the use of expression trees in EF, but I also wanted to highlight their extensive use in the somewhat recently open-sourced Reaqtive project. This project provides a platform for hosted reactive operations allowing for stateful stateful checkpointing and recovery of standing and active subscriptions. It provides libraries for serialization of expression trees (allowing for queries and their components to be saved in stateful storage while at rest), which is huge in itself. However, the implementation of and extendibility of the observers and observables could be made dramatically simpler if .NET had support for async expression trees, so I'd ask that these be considered in the next overhaul of expression trees, whatever shape that takes. I understand the argument in here for breaking old dependencies on existing expression trees, but I don't necessarily see why this isn't something that is run in the .NET Core style of move-faster-likely-breaking-things as things transitioned from .NET 4 to .NET 5. Given that the implementation of each of the proposed changes is thought to be pretty straightforward and because Bart has already done some exploration in what might be needed to support different branches of support with the ExpressionTreeLike types and even has a sample implementation of this, why can this not simply be something that is opt-in, similar to that of the nullability flag in csproj wherein developers can choose to use the legacy expression tree (selected by default) or can opt into a more modern expression tree that will continually evolve as C# does. I don't know to what extent .NET is able to do feature flags, as it were, in the code, but what if modern expression support were an opt-in on a feature-by-feature basis? Today, my .csproj contains: <PropertyGroup>
<Nullable>enable</Nullable>
</PropertyGroup>Why can't modern expression trees be similar? This would enable all available modern expression tree functionality (similar to setting language support to vNext in that it's likely something will break with new changes). <PropertyGroup>
<ModernExpressions>true</ModernExpressions>
</PropertyGroup>And especially since there's such a laundry list of individual features, allow them to be opt-in by name: <PropertyGroup>
<ModernExpressions>
<NullCoaselscing>true</NullCoalescing>
<Async>true</Async>
<Dynamic>true</Dynamic>
</ModernExpressions>
</PropertyGroup>Those features that are inextricably linked to all of them are included just by enabling ModernExpressions (doesn't happen often, but does happen per the opt-in contract). But this means you don't have to enable all the new features all at once in a single release and trivially opens the door to revisiting to add new expressions features in a way that people can use if they want or simply opt-out of. At launch, there's zero net impact until developers decide to proactively opt into the new functionality - a great many will, but if their application breaks, I suspect they'll be more reasonable about the migration. After launch, there's some impact as developers upgrade packages indiscriminately, but the analyzer saves them from themselves here. And in the longer-term, new applications get to enjoy rich expression tree support and older applications keep on using the legacy approach. |
|
It's strange that something as important as this is not even being discussed. What's blocking this?! |
Nullability has absolutely no runtime impact. That's precisely why it was ok to go that route, and it's a strong reason why it could be added without being impactful to the ecosystem. That's very different from any approach that could actually end up impacting anything at runtime. :( |
|
That's a good point, but at the same time, it begs the question of whether the future of C# development is just syntax updates devoid of any runtime changes for the same underlying concern of breaking some ancient backwards compatibility somewhere. I was reading through the proof of concept last night to get a better understanding how it it's implemented and the changes to Roslyn itself don't appear to be that substantial which led me to the following. Your concern is that by changing the runtime for expressions, it could have disastrous and unknown impacts to developers consuming that functionality. The approach used in the proof of concept appears to be one that allows for over-time extensibility so as to avoid this point-in-time feature lock, but implementing that approach requires breaking the existing language runtime contract. The best path forward here then is to just leave existing expression trees exactly as they are indefinitely moving forward and chalk it up to a poor design decision in early C# days. But then why can't you then implement the newer, extensible expression trees with the bare minimum runtime support necessary under a different namespace removed from the domain of the language team? I anticipate you saying that this just moves the needle from C# 3.0 feature lock to C# 11.0 - fair, so why not shift the actual implementation of these modern expression trees out to a separate package. The .NET library ecosystem is broad and rich and has a readily understood contract in place regarding versioning, lifecycle and modernity expectations. The language ensures backwards compatibility to a back-breaking level - I get that, but separate implementation packages don't generally have nearly the same concern for backwards compatibility. But further, if I need what's in v1.0 and v2.0 breaks my implementation, I need only install v1.0 and compatibility is on me as the developer to ensure for a safe upgrade path. Could this be a more acceptable path forward?
Thanks for the ongoing consideration! |
Yes. We work steadfastly to not make runtime changes, except in very narrow circumstances, and when teh net benefit will likely be enormous, and highly unlikely to break anyone. For example, we are considering changing how exceptions are thrown in async/await. This should not be noticeable at runtime, unless doing very specific groveling of the low level async/await machinery. Similarly, we may allow a However, the above is very different from expression trees. |
I don't see any reason why we could not do that. I may have missed it, but did someone give a reason why that would not be possible? :)
I would not say this would have C# 11 lockin. Indeed, if doing this again, i would model around Roslyn, with the same promises (or more specifically, the lack thereof :)) around compat. Specifically, i would say that you should expect breaks, and that as the language revs, this API will move to match it, and it's incumbent on the consumer to deal with that (the same way we deal with roslyn changes today).
Definitely :) |
|
Circling back here.. if simply using a different namespace for Expressions v2 is a viable path forward for modernizing it, what are the remaining blockers, if any, to moving forward? Is there anything I can personally do to help shuffle this along apart from merely advocating for it? |
|
Found this discussion while writing yet another EF query that required me to do .Select(x => new Dto {
Foo = x.Foo == null ? x.Date : x.Foo
})instead of the sane thing, .Select(x => new Dto {
Foo = x.Foo ?? x.Date
})and I'm guessing I'll have to use it a while longer still |
|
That's just the coalesce operator, which has always been an expression tree node type. I'm guessing that's just a failure on the EF provider to not understand how to translate the coalesce operator into the SQL COALESCE function. |
|
This comment — dotnet/efcore#33336 (comment) — mentions that it is a limitation of expression trees. That said... I was looking for instances of the issue in my codebase to show real-life examples, and I'm having trouble finding any. So either I hallucinated it, or it's not actually the issue I had in mind. I'll pop back into this discussion should I actually find something |
|
|
|
Having just re-read this entire thread, all the while carefully considering the concerns and explanations provided by the likes of @CyrusNajmabadi, @333fred, et al. I must say they are making about as much sense as how many "new" C# syntactic features have had support added for them in expression trees over the last decade or so, which is, of course, a glorious zero. Let's consider this comment by @333fred — as I think it succinctly and clearly presents the supposed justification:
(emphasis mine) Sounds reasonable, right? Except that it should take no more than 5 seconds for anyone remotely familiar with C#/.NET's and the surrounding ecosystem's progression over the last couple of years, to realize two things, which immediately obliterate this reasoning: Swap the terms "new node" with "new built-in anything" and you'll get an equally sound argument for never adding any feature at any level in any area to the framework/standard library ever; because "what about existing tools that won't immediately support it?", I guess. It is unbelievably unconvincing. There is literally a fundamental analogy here between any hypothetical new expression nodes, and any new standard constructs in general — with the most notable recent example being what happened with Notice that according to the same logic as the one that we're being provided here as to why expressions are stuck at the C# of 10 years ago, apparently the addition of
It gets even worse, this argument is, in fact, even LESS applicable in the case of expression trees, because all consumers of expression trees (most prominently EF), ALREADY ONLY support a subset of "all possible expression trees". An expression tree consumer not supporting a certain thing at a given time is already the status quo, and as such, fully expected. Funny. Some people in this thread (e.g. @olmobrutall here) tried to point out the same thing, heck, even some comments on related Stack Overflow questions made this point, and made more basic logical sense than anyone from the other side here. I'm just trying to help reveal the folly in this reasoning more clearly and bring more awareness to the issue. |
No. SyntaxNodes are just raw syntax. They do not contain important, necessary, semantics (like "what on earth does this |
But i think it does - new code structures can't be compiled using old roslyn's, they would return a syntax error. The compiler is also not constrained to emit the same byte code from version to version, only that executing the bytecode comes with the same outcome (eg, i think how errors are handled in async can produce different bytecode in different roslyn version, but no effective difference in outcomes other that performance considerations) An old AST might have a switch (node N){ In this scenario, the source code hasn't changed, and the destination framework hasn't changed, just a library somewhere in the middle of a stack getting a version bump which "only" contains optimisations could be enough to induce runtime AST crashes. There is also the development cost - issues with Roslyn interpretation are caught with red squigllies as you type. AST's are pure interpretation, so you only find out you've done a 'silly' at runtime, potentially quite a bit later in the dev cycle. |
|
Analyzers and/or compile-time translation of the expressions might help detecting what unsupported expressions. |
What expression trees is a LINQ provider going to accept is a grey area already. Some mini-LINQ providers maybe only accept MemberAccessExpression because they are only meant to refer to properties (like strongly-typed reflection) while others could support almost everything. I can see that is tempting for you to be able to restrict this at compile time with some versioning or a massive attribute enumerating the supported ExpressionNodeTypes but it is very complicated, think of this query: The Even without considering new nodes, the developer already has to know which methods and properties are supported by the LINQ provider and what not, this is a much bigger gray area anyway. So… just not solve this communication issue. LINQ to expressions base some pitfalls
Roslyn nodes are full fidelity, so I can see there is tension a versioning issues. Expression trees on the other side are more abstract and easily consumable:
Also, the API validates at run-time the types of the arguments etc quite strictly, so there is not a lot of freedom to make another valid expression tree really. I can only think maybe inserting/removing some unnecessary castings or a different method overloads resolution (already happening?) Could you make an example of c# code that could produce two different valid expression trees and would benefit from a change?
The original spirit of the expression tree api was to implement like 40% of the language (only expressions) to provide 90% of the value when translating to SQL, and I think this was very wise. Maybe there are some use cases for a new SuperExpressionTrees that supports statements and all the new features (shaders?, auto-derivation for tensors computations?) but the change will be massive and not backwards compatible, so if it ever gets done should be an different API and not an expansion of the current one. But if LINQ was designed with the same spirit but today I think they would have increased the amount of language supported just a little bit more:
Query<Person>().Select(a=>a.Car?.RegistrationDate)but this is not supported.. so you have to write Query<Person>().Select(a=>a.Car!.RegistrationDate) // works… except of the Car is null, then you get a cast exception from null to DateTimesome other additions that don’t even require new exprsssion nodes:
I need an example to see why is this so complicated, the current compiler is able to translate Roslyn nodes into old expression trees. I don’t think minimally increasing the target nodes would be so hard but an example would help.
As I mention, knowing what is supported is very complicated and the list of supported members is more important than the list of nodes. A Roslyn analyzer is probably a much better solution than versioning at the compiler level… but is very challenging because the If you all can accept the idea of implementing a small increase of the current expression api, instead of a big change for a more ambitious but risky change I’m willing to prepare a PR. I made it already in C# 5 or 6 back in codeplex times |
|
It's been a minute since I looked at it, but there's already a fork of Roslyn that's been outfitted to support all the language changes up through C# 11 from Bart de Smet here. I'm going to have to read through it again and compare to the various points of feedback above as it doesn't use a version number and otherwise works around the System.Linq limitations. |
|
This could be done with source generators and interceptors, I assume. |
|
@CyrusNajmabadi How would it expose your internal model? I’m suggesting exposing the raw tokens, not the semantic model. There’s no such thing as “member access”, just a period token. Perhaps, initially, the compiler could ensure it’s valid syntax. As @erik-kallen said, EF consumers could either consume them at runtime or use source generators to generate code at compile time. That would easily lend itself to macros using source generators (where the compiler wouldn’t enforce syntax validity). |
|
If it's the raw tokens then it's the concrete syntax model, not the abstract one. |
|
And why shouldn't that be exposed? Other languages manage just fine with exposing token streams. |
You are fundamentally asking for somethign else now. And we don't expose something as major as that just off of one person asking for it, with exactly zero consumers saying it's what they actually need over expr trees. |
|
Hello, just wanting to make sure I'm on the same page, since there are seemingly two different discussions going on. Forgive me if this is re-hashing the old points again, but I feel this might help more than me. There are fundamentally two different parts to this discussion. Things Currently Supported By Raw Expression Nodes, But Not Supported In Lambda ExpressionsThis includes things like CSharpExpressions was already mentioned as an example of solving some of those by implementing new nodes, but allowing most nodes to be reduced to a more complex tree using existing expression nodes. However, another example which does get used in production is LINQKit. Because it's the only way I can easily call a typed lambda expression from another lambda expression. Things Not Currently Supported By Raw Expression NodesThese are things which would explicitly require new node types to work. An example noted by CSharpExpressions is below.
My Issue / QuestionSomeone please correct me if I am misunderstanding, but I don't understand the controversy of fixing the first issue. The only possible downside is it would allow someone to more easily write expressions which contain some node types not supported by all query transformers. However, it is possible today to write those same not supported expressions today. They just have to be done by hand. Meanwhile, the convenience of the null-coalescing operator ( Plus, without LINQKit everything either has to be in one large lambda, violating almost every code readability standard out there, or everything which calls something else must be a hand constructed expression. Which is something most C# programmers, including the rest of my team, aren't familiar with and can't work on. Would someone please explain to me why fixing this first issue is so controversial. |
|
You should just rip off the bandaid and start pushing updates to the API. If you want to play it safe - do it on a small surface area of new language features to give the community / providers time to update for compatibility. Then on the next C# release push out the next wave of updates once library authors get the gist of what’s to come. Stop overprotecting the community though. We will pick up the pieces. PR’s will be raised, compatibility issues will be fixed. In MUCH less time than it takes you guys to debate all the downsides of how to minimized breakage. Let’s get on with it already! |
|
Last I heard, backwards compatibility will remain a priority because big players are paying big money for support, and breaking their .NET Framework 3.5 projects would turn big money into slightly-less-big money as they move somewhere else. Us, little ones, we just enjoy the crumbs off the table. |
|
Anyone on .NET Framework is not affected by changes in .NET Core 1-3 or .NET 5+ |
Yes and no. As someone who deals with both, you'd be amazed at how much stuff still works with Framework. In this case, the only fallout will be yet another thing that can't be copied and pasted into a decade old code base. |
|
I honestly believe this feature is much more needed than Span or Memory since most of big players already manage memory properly where its needed, but runtime code generation is still missing, which is a blocker for many |
|
Highly disagree. I, like most expression tree users, would love to see expression tree evolution, but spans are a god-send when it comes to performance. Managing memory prior to spans was a massive PIA. Want to do anything with a substring or subarray? That’s an allocation you (almost never) can avoid. Sure, |
|
We write an LLM train api, needless to say we process hundredrs of gigabytes of text, and we find StringBuilder and raw string access to be way more effective - and most impotant - way more simple to write and read - than using spans. |
|
@AlexeyTliss Your use case is one of millions of customers we have to consider. It's fine that you think this is more important for your own needs. But that doesn't mean the same holds true for the ecosystem as a whole. |
|
@CyrusNajmabadi , but let's be honest, expressions are lagging behind C# features like hell. =( |
|
I'd love to voice support for an evolution of expression trees as well. Since it was mentioned the depriorization of work on this is due to not meeting the bar for investment/not having enough customer relevancy compared to other features, I thought I'd just throw in my anecdotal experience. In 4 quite different types of jobs that I've worked in over the past 15 years or so, I always ended up working on something where using expression trees was extremely beneficial, if not borderline inevitable. And I should say, none of these use cases were at all related to EF Core (just mentioning as my impression is much of the discussion - understandably - seems to gravitate a bit around Entity Framework. But there are so, so many other great use cases too!). My typical use case is almost always performance related. As random examples, my last use cases, both from within the past few weeks alone: So oftentimes stuff that is typically highly reflection based, but needs to be made significantly more efficient for some type of repeated invocations. The big issue for me is that there isn't much of an alternative really. Sometimes you can work your way around compiling a new assembly on the fly via Roslyn and loading that up. I've done that for another recent project. But in my view, it's not a great replacement in many cases - personally,
As for the elephant in the room, i.e. compatibility concerns: I definitely understand those in the context of query providers (at least somewhat; albeit last time I used EF, many many years ago, only some expressions were supported anyway, so you never quite knew what worked or not until runtime - did that change?). Long story short: Support for modern features in expression trees is sorely missed! For my personal work (!), this would be a much more beneficial than most other ongoing language features at the moment, so I'd definitely be really excited if those were revisited! |
It is absolutely like that! And for the features that do "work", you also need to check what SQL they will generate. |
|
Linq recently got some new features (LeftJoin and RightJoin) hence the argument made in this comment may not be a good reason anymore to avoid any changes in the library. |
Everyone has their own views on what the main issue should be :) |
Side note:
A very interesting library that actually does this (in a somewhat specific scenario) is https://github.com/koenbeuk/EntityFrameworkCore.Projectables/ It achieves this by implementing:
For example, it supports using System;
using System.Linq.Expressions;
using EntityFrameworkCore.Projectables;
using EntityFrameworkCore.Projectables.Extensions;
Expression<Func<int?, int>> expr1 = (x) => x ?? 0;
Expression<Func<Example, int?>> expr2 = (x) => Example.OptionallyGet(x);
Console.WriteLine(expr1);
// x => (x ?? 0)
Console.WriteLine(expr2);
// x => OptionallyGet(x)
Console.WriteLine(expr2.ExpandProjectables());
// x => IIF((x != null), Convert(x.Field, Nullable`1), Convert(null, Nullable`1))
public class Example {
public int Field;
[Projectable(NullConditionalRewriteSupport = NullConditionalRewriteSupport.Rewrite)]
public static int? OptionallyGet(Example x) => x?.Field;
}I think this is a very interesting approach and investigating it further could be very effective, as it allows each "consumer" of the expressions to implement a slightly different translation from C# to their expressions. For example in the case of pure expressions, repeating a subexpression (as done by EntityFrameworkCore.Projectables) is just fine, while on side-effectful expressions it would certainly be a bad idea. This approach can also provide compile-time diagnostics about issues in the expression translation; for example fails with the following error marked on
Another reason why I find this interesting is that this also opens the door to converting C# ASTs to any other syntax, regardless of what can be represented in The major disadvantages I can see regarding such an approach are:
Even with these significant issues, I think this is a direction that might make sense to investigate and does not necessarily require effort within the compiler, but can instead be implemented with the existing tooling. |
This is an error that got this morning:
I think I have received around 200+ similar errors in the last years. Almost always related to using Just to mention that "waiting forever" also produces exceptions in production that depend on data (the query returns null). While adding support
Can I get green light to attempt a PR? Pleeeeease |
|
Let's just flesh this one out and get everyone on board with the general direction first.
|
|
Continuing from this, it's possible to do it with csharplang/roslyn involvement only. Consider this helper method: public static B ConditionalAccess<A, B>(this A? receiver, Func<A, B> accessor) => receiver == null ? null : accessor(receiver);Whenever a compiler sees a ?. it emits an Expression.Call if a regular/extension method with this name/shape is present (a lot of stuff e.g. foreach, await, deconstruction works this way so this is not a new concept). This method could be provided by the end-user, some library/nuget or even generated on-demand by roslyn. The important thing is that it has a well-known name/namespace so that providers can detect and special-case it. The main drawback here is that this pattern only works for additions that can be expressed as a method call so no out var/await/etc. |
|
Perhaps a library-only solution using interceptors could work? Something like and a source generator could intercept the call and generate appropriate factory method calls? |
|
@bachratyg that does not compile because it is still directly relying on the quoting performed by Roslyn. Instead, this compiles: using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
var src = new string?[] { "asd" }.AsQueryable();
var q = MyExpression.Make(() =>
from a in src.AsEnumerable()
where a?.ToString() == ""
select a
);
var q2 = src.Make(s =>
from a in s
where a?.ToString() == ""
select a
);
static class MyExpression
{
public static Expression<T> Make<T>(T wrapped)
=> throw new NotImplementedException();
public static Expression<U> Make<T, U>(this IQueryable<T> source, Func<IEnumerable<T>, U> wrapped)
=> throw new NotImplementedException();
} |
That's just to have some variable to build on. In prod it's likely a DbSet or some IQueryable method argument. |
|
The problem, as pointed out by both me and @ranma42, is that the IQueryable methods take an Expression<Func> parameter. The sourcegen will have to work on IEnumerable instead, whose extnsion methods instead take a Func |
|
@erik-kallen there's no question about why this needs to be done. I have concerns about how to do this so the result is good enough. Boilerplate-wise this: var q = MyExpression.Make(() =>
from a in src.AsEnumerable()
where a?.ToString() == ""
select a
);is no improvement over this: var q =
from a in src
where a.Elvis(s => s.ToString()) == ""
select aThe latter does not even need sourcegen and all the trouble it brings. |
|
No, there is no way to get a conditional operator into a quoted expression. Whether it is an improvement is in the eye of the beholder. And the .Elvis() can't be translated to SQL anyway. Also, a sourcegen solution could theoretically support |
|
Maybe I can add some context here. We did make a big change to expression trees in recent memory: the move to Roslyn. The compiler is responsible for encoding the tree represented in expression trees, and when we rewrote the compiler to C# in Roslyn we rewrote the implementation for the tree. This was tricky because the native compiler actually just dumped the in-memory tree into the expression tree. Roslyn doesn't use the same tree, so we had to map from the new Roslyn trees to the old native trees. We did a pretty good, but not great, job. There were a number of bugs around different nodes being generated for the same source. This is how we discovered that providers are very sensitive to the exact shape of the tree. There were many bugs filed around tree differences. At first we tried to fix those. However, it didn't take long for us to get a new set of bugs about how "fixing" the tree broke their provider since they had adapted to the new tree. This was made worse by the fact that dumping implementation details (the native compiler's tree) into public surface area was not good. Some of the trees were almost impossible to generate well from Roslyn. And some of the trees were just incorrect, because the native compiler had a kind of buggy or incorrect internal representation of a particular syntax. Roslyn has this new thing that is a public, supported, designed surface area for semantic nodes ( Roslyn has also added a bunch of source generation features that overlap with expression tree scenarios. They also address some big problems with expression trees. For instance, expression trees allow you to express things that would be illegal in C#, like violating accessibility. While this is useful, it's also problematic. By generating source that split is avoided. It also enables Native AOT support since certain expression trees require dynamic code compilation to execute, and all expression trees have to be interpreted otherwise. Source generation usually creates compatibility around build time, not run time. There are limitations to source generation but also benefits in that you usually figure out the problems when you recompile, not when you update the .NET runtime in production. Lastly, there's the matter of testing. While the test suite isn't bad, it's not to the level of completeness that would catch all possible breaks, as can be seen with the previous fixes. That makes changes riskier and more expensive. All together, the state of expression trees is a challenging environment to make changes in. While improvements could provide benefit, any change would have to carefully weigh cost and risk to be accepted. |
Agreed! https://github.com/EFNext/ExpressiveSharp |
|
Would it be correct to say that a lot of the problems with keeping expression trees up to date with language changes stem from them being designed and implemented before Roslyn and Roslyn's compiler API existed? Has there been any discussion of a parallel, side-by-side, non-backward-compatible v2 of expression trees, that would be closely tied to the Roslyn surface area in such a way to make it sustainable (i.e., new features added to the language would be added to expression trees "for free")? Would such a thing even be possible in theory? |
|
No, this is not related to Roslyn. It is based on the fear of breaking existing consumers if new nodes are introduced or we change how existing operations are encoded. |
|
Does this comment still apply given all the recent additions? Like right join or full join? |
|
For context, LDM approved adding |
|
If this is all about breaking changes, can't we just have an assembly level attribute or something in the csproj that states "UseLegacyExpressionTreeParser" for backwards combatibility? If its's something that could be optionally applied even lower (class/method) then we could use a mix of legacy vs new technology. Tbh, this attitude of "we can't move forward because it will break consumers" doesn't seem to make sense when basic things like null propagation in lambdas is such a massively sought after feature. (It's also worth noting that probably 90% of the people coming across this issue just want null propogation in their expression trees - adding that alone would bring so much benefit, let's not make 'perfect' the enemy of 'good' here) :) |
It's unclear what that attribute woudl be expected to do.
This sentence doesn't logically follow for me. Something can be sought after, but still be less important than the risks around breaks. |
It would be expected to make the behaviour of the expression trees in that assembly/class/method to be like the legacy behaviour (what it is now) rather than the new behaviour which might have some breaking changes for consumers
Agreed, but what I'm saying is that there are ways of mitigating those risks (I've suggested one - a flag to use legacy behaviour), and the importance and value of meeting consumer demand here should not be underestimated in evaluating the cost benefit situation. So many secondary libraries are constantly caused problems by Lambda's lack of support for null propogation, it prevents massive amounts of meta functionality from being implemented. For example the library I'm working on at the moment wraps assertions to extract meta information, but the act of turning them into lambdas is changing their behaviour, in 90% of cases it's simply null propogation that is the culprit. |
i don't see how this helps the breaking change. even if you opt in, the libraries are still broken. |
|
Can you help me understand what you mean? The attribute would cause a build behaviour which would mean that the processing of those libaries in those assemblies/classes/methods would behave like the legacy behaviour, so the libraries would work correctly - am I missing something? |
|
I like this idea. A flag which lets you opt in to all the new goodness per project, and accept breaks if any of the libraries you're using don't understand the linq queries in the current project. If you hit a problem, you undo the flag and the project continues compiling exactly like today and you have to undo the LINQ goodness. The control is entirely in the user's hands, and no one better than the user can decide whether the user needs to worry about breaks or not. |
|
Guys it's 2026. Previously unsolvable problems like this one have drastically different tools available to solve. Why doesn't the .NET team ship a Expressions Trees migration agent, and why don't you perform a Github-wide analysis of all libraries that would break, and why don't you generate a PR for the top libraries with the said migration agent and submit it to the authors for review as part of a pre-migration phase. You don't need to block on innovation for literally decades due to being bound by old norms. This is obviously so an agents-solvable problem, since it's a focused specialized skillset that can be dog-fooded on numerous internal libraries and tested out on so many public open-source codebases, then easily rolled out to the masses. Can release a one-click to install agent skill with a landing page for library owners showing them a video of how they can issue the agent to run on their codebase. Let's get with it team! |
I would not generally expect this to be possible. Though maybe someone could write a helper lib there. |
|
What blockers do you foresee? And we even already have a handy visitor. |
|
Testing cost and the extra burden off now having to ensure this translation compat layer exists and doesn't break. It's another set of behaviors to have to maintain. And because it would be to support old and brittle providers it's would have the same difficult compat challenges that affect ETs today. I would personally push against it because I think it would be too burdensome for the new API and would limit being able to effectively change it over time. |
|
I suppose the translation layer could just as well be a community effort available on nuget.. This would free up the c#/dotnet team to purely focus on the new ET api. |
|
That would work for me. Different libraries can also have different support expectations. So a community one could have different accepted levels of quality/support/compat/etc. |
|
I see the main limitation of expression trees as mixing concerns between executable code representation and query definition. Expression tries to represent both executable .NET code (Compile()) and query semantics, while most query providers only need a semantic representation. If Roslyn provided a mechanism to translate bound lambdas into library-defined expression-like types, libraries could decide which language constructs they support and how to represent them. The compiler would still handle parsing, binding, overload resolution, and diagnostics, while providers would only define translation into their own query model. This could let providers support modern C# features incrementally without being constrained by System.Linq.Expressions. |
|
Yes a problem with LINQ to SQL/to EF/etc. and IQueryable has been there's no compile-time checkable/toolable way for query providers to represent which language constructs they do and don't support. I feel like an incompatible v2 of expression trees probably wouldn't be worth doing unless it addressed this problem, seems like a difficult problem though. |
|
yes, there can be an automatic v1 to v2 translation, but never the reverse, without a possibility of a crash at runtimethat the compiler can't even warn about. |
|
This sounds very reminiscient of the arguments made back in 2008 when Python 3 was released, and we know how well that went. |
|
@theunrepentantgeek, you mean migration between providers' expressions? Today, it can be done with agent skills like migrate-xunit-to-xunit-v3 skill. And because the translation happens at compile time, it can be considered safe. |
and others
Uh oh!
There was an error while loading. Please reload this page.
Expression trees today support a limited subset of expression lambdas. This issue is to track the request to extend them to support more or all of the language constructs that can be expressed in a lambda.
A prototype for work in this direction can be found on https://github.com/bartdesmet/ExpressionFutures/tree/master/CSharpExpressions and https://github.com/bartdesmet/roslyn/tree/ExpressionTrees (based on a fork of Roslyn a while ago).
/cc @bartdesmet
All reactions