Was Shia LeBeouf's appearance at the Berlin Film Festival with a paper bag over his head ('I'm not famous' written across it), a publicity stunt? Was he a stand-in for the troublemaking antics of Lars von Trier himself? It seems the sort of thing the Danish filmmaker himself would do, except that he's still in a self-imposed vow of festival and media silence after telling a Cannes press conference he understood Hitler.

The funny thing about von Trier is he's never had to resort to tricks and gimmicks to get attention - unless he truly does want to treat the cinema firmament with contempt, that is. He's made some of the most distinctive and incisive movies about the hypocrisy of American culture (Dogville), healing (Antichrist) and depression (Melancholia) and he's always been the one to delight in peeling back layer of false gentility.

There's also something about von Trier in that he seems to be the only director who could get away with making a movie about a self-described Nymphomaniac, Joe (Charlotte Gainsbourg) without it sounding like something you'd expect to watch after midnight on some pervy, creepy pay TV channel.

If a director came out of Hollywood saying he wanted to do a serious exploration about the effects of an insatiable sexual appetite on a woman, we'd all snigger and just wait a movie about a woman as sexually obsessed as most middle aged Hollywood producers are.

But von Trier confounds expectations as well as everything else. There's no Adrian Lyne-inspired neon lights throwing shadows on shimmering bodies through thin Venetian blinds. We meet Joe lying in a dingy alley after apparently being bashed. When Seligman (Stellan Skarsgård) seeing her laying in a bloody heap, he picks her up and helps her back to his apartment to clean her up and call a doctor.

Joe insists she just needs rest so as Seligman makes her tea and the conversation veers off in some strange directions like his love of fly-fishing, she tells him her story. Starting with discovering her vagina and the sexual pleasure it offered at a very precocious, pre-pubescent age, Joe embarks upon an odyssey of fulfillment-seeking behaviours and encounters.

It starts with a competition between her and her best friend as young teens to win a bag of sweets by shagging the most men on a train, and by the time Joe's a young woman (played by the frequently-naked Stacy Martin) she's somehow managing up to ten lovers a day. In true movie fashion it doesn't say much about how she has time for sleep or work, even though she has a job as a secretary at a print shop.

It's also not clear if Joe's really just seeking love despite trust or intimacy issues, has a mental illness or is abusing drugs or alcohol (all said to be causes of hypersexuality behaviours in real life) – she seems to just want as much sex as she can get, and even though she seems to enjoy it at the time, it doesn't seem to fulfill her.

There's a strong clue in the backstory about her loving father (Christian Slater), a man who taught her a love of nature and died early, but it's not treated as an obvious cause to Joe's current state of mind.

Maybe von Trier – who's always been a master at layering meaning like tiers on a cake – is even subverting our expectations about what a movie about a nymphomaniac should be about – it's certainly not the nymphomania of porn and Hollywood. In one scene Joe gets rid of a lover by giving him some reverse psychology, telling him it's over because he'll never leave his wife (the last thing she really wants him to do).

With the next beau about to turn up, the guy she's just got rid of lands back on her doorstep, suitcase in hand. He's not only left his wife (Uma Thurman), she's followed him to Joe's apartment and bought their children. As Mrs H, Thurman is an acerbic delight, all sweetness and light at her husband and Joe while the acid of her rejection bubbles just behind her eyes ('would it be all right if I showed the children the whoring bed?' she asks gaily). Another scene shows the bedridden Christian Slater being cleaned up after soiling himself, not usually the stuff of soft porn.

Many of the scenes, particularly the ones between Joe and Seligman, are more like a play. The dialogue and characterisations are very tightly constrained, almost cyphers to tell you the chapters of Joe's story. She even talks about her life as if it's a novel that can't be rushed, the details of the journey more important that the destination.

One fair criticism of the film is a disappointing view of men. Usually movies are guilty of endlessly narrow mother/whore portrayals of women, something von Trier's never been guilty of. But men get short shrift here, all too ready to take any advantage they can for pussy. Even a seemingly decent middle-aged married man on his way home to his wife rejects the teenage Joe at first but eventually gives in to a 30-second blowjob in a train carriage.

Another mystery is why Joe decides to tell a complete stranger her sexual history at a time of apparent vulnerability and distress. Shia LeBoeuf's English accent is atrocious, but he's worth seeing – after his recent plagiarism scandal and increasingly abberrant behaviour it might be the last chance you have to see him act before he disappears in a very public, Lohanesque flame-out.

It's fairly explicit by cinema standards, but the film highlights an interesting thing about the time we live in. Before VCRs and the Internet, a movie like this would have been the only place boys could get a glimpse of nudity and it would have been a very sought-after underground hit. 30 years ago that would have played a very large, unmentionable part of the economics of the movie.

Today, with hardcore porn everywhere at a click or swipe, not even the filthiest scenes in Nymphomaniac will be enough to drive the film's performance through voyeuristic curiosity – even if that really is Shia LeBeouf's full frontal erection you think you see for a second as Stacy Martin prepares to straddle it.

Like in most European arthouse movies, there's a lot here that doesn't make sense. Maybe it doesn't make any sense to von Trier either, and he just wanted to see show Stacy Martin naked as long as he could. But he's always been cleverer than just the desire to shock, and even if there are parts you don't get, he'll still one of the most interesting directors working today.