Open File Dialog MVVM

The best thing to do here is use a service.

A service is just a class that you access from a central repository of services, often an IOC container. The service then implements what you need like the OpenFileDialog.

So, assuming you have an IFileDialogService in a Unity container, you could do...

void Browse(object param)
{
    var fileDialogService = container.Resolve<IFileDialogService>();

    string path = fileDialogService.OpenFileDialog();

    if (!string.IsNullOrEmpty(path))
    {
        //Do stuff
    }
}

Long story short:
The solution is to show the OpenFileDialog from a class, that is part of the view component.
This means, such a class must be a class that is unknown to the view model and therefore can't be invoked by the view model.
The solution of course can involve code-behind implementations as code-behind is not relevant when evaluating whether a solution complies with MVVM or not.

Beside answering the original question, this answer also tries to provide an alternative view on the general problem why controlling a UI component like a dialog from the view model violates the MVVM design pattern and why workarounds like a dialog service don't solve the problem.

1 MVVM and dialogs

1.1 Critique of common suggestions

Almost all answers are following the misconception that MVVM is a pattern, that targets class level dependencies and also requires empty code-behind files. But it's an architectural pattern, that tries to solve a different problem - on application/component level: keeping the business domain decoupled from the UI.
Most people (here on SO) agree that the view model should not handle dialogs, but then propose to move the UI related logic to a helper class (doesn't matter if it's called helper or service), which is still controlled by the view model.
This (especially the service version) is also known as dependency hiding. Many patterns do this. Such patterns are considered anti-patterns. Service Locator is the most famous dependency hiding anti-pattern.

That is why I would call any pattern that involves extraction of the UI logic from the view model class to a separate class an anti-pattern too. It does not solve the original problem: how to change the application structure or class design in order to remove the UI related responsibilities from a view model (or model) class and move it back to a view related class.
In other words: the critical logic remains being a part of the view model component.

For this reason, I do not recommend to implement solutions, like the accepted one, that involve a dialog service (whether it is hidden behind an interface or not). If you are concerned to write code that complies with the MVVM design pattern, then simply don't handle dialog views inside the view model.

Introducing an interface to decouple class level dependencies, for example an IFileDialogService interface, is called Dependency Inversion principle (the D in SOLID) and has nothing to do with MVVM. When it has no relevance in terms of MVVM, it can't solve an MVVM related problem. When room temperature does not have any relevance whether a structure is a four story building or a skyscraper, then changing the room temperature can never turn any building into a skyscraper. MVVM is not a synonym for Dependency Inversion.

MVVM is an architectural pattern while Dependency Inversion is an OO language principle that has nothing to do with structuring an application (aka software architecture). It's not the interface (or the abstract type) that structures an application, but abstract objects or entities like components or modules e.g. Model - View - View Model. An interface can only help to "physically" decouple the components or modules. It doesn't remove component associations.

1.2 Why dialogs or handling Window in general feels so odd?

We have to keep in mind that the dialog controls like Microsoft.Win32.OpenFileDialog are "low level" native Windows controls. They don't have the necessary API to smoothly integrate them into a MVVM environment. Because of their true nature, they have some limitations the way they can integrate into a high level framework like WPF. Dialogs or native window hosts in general, are a known "weakness" of high level frameworks like WPF in general.

Dialogs are commonly based on the Window or the abstract CommonDialog class. The Window class is a ContentControl and therefore allows styles and templates to target the content.
One big limitation is, that a Window must always be the root element. You can't add it as a child to the visual tree and e.g. show/launch it using triggers or host it in a DataTemplate.
In case of the CommonDialog, it can't be added to the visual tree, because it doesn't extend UIElement.

Therefore Window or CommonDialog based types must always be shown from code-behind, which I guess is the reason for the big confusion about handling this kind control properly.
In addition, many developers, especially beginners that are new to MVVM, have the perception that code-behind violates MVVM.
For some irrational reason, they find it less violating to handle the dialog views in the view model component.

Due to it's API, a Window looks like a simple control (in fact, it extends ContentControl). But underneath, it hooks into to the low level of the OS. There is a lot of unmanaged code necessary to achieve this. Developers that are coming from low level C++ frameworks like MFC know exactly what's going on under the hoods.

The Window and CommonDialog class are both true hybrids: they are part of the WPF framework, but in order to behave like or actually to be a native OS window, they must be also part of the low level OS infrastructure.
The WPF Window, as well as the CommonDialog class, is basically a wrapper around the complex low level OS API. That's why this controls have sometimes a strange feel (from the developer point of view), when compared to common and pure framework controls.
That Window is sold as a simple ContentControl is quite deceptive. But since WPF is a high level framework, all low level details are hidden from the API by design.
We have to accept that we have to handle controls based on Window and CommonDialog using C# only - and that code-behind does not violate any design pattern at all.

If you are willing to waive the native look and feel and the general OS integration to get the native features like theming and task bar, you can improve the handling by creating a custom dialog e.g., by extending Control or Popup, that exposes relevant properties as DependencyProperty. You can then set up data bindings and XAML triggers to control the visibility, like you usually would.

1.3 Why MVVM?

Without a sophisticated design pattern or application structure, developers would e.g., directly load database data to a table control and mix UI logic with business logic. In such a scenario, changing to a different database would break the UI. But even worse, changing the UI would require to change the logic that deals with the database. And when changing the logic, you would also need to change the related unit tests.

The real application is the business logic and not the fancy GUI.
You want to write unit tests for the business logic - without being forced to include any UI.
You want to modify the UI without modifying the business logic and unit tests.

MVVM is a pattern that solves the problem and allows to decouple the UI from the business logic i.e. data from views. It does this more efficiently than the related design patterns MVC and MVP.

We don't want to have the UI bleed into the lower levels of the application. We want to separate data from data presentation and especially their rendering (data views). For example, we want to handle database access without having to care which libraries or controls are used to view the data. That's why we choose MVVM. For this sake, we can't allow to implement UI logic in components other than the view.

1.4 Why moving UI logic from a class named ViewModel to a separate class still violates MVVM

By applying MVVM, you effectively structuring the application into three components: model, view and view model. It is very important to understand that this partitioning or structure is not about classes. It's about application components.
You may follow the widely spread pattern to name or suffix a class ViewModel, but you must know that the view model component usually contains many classes of which some are not named or suffixed with ViewModel - View Model is an abstract component.

Example:
when you extract functionality, like creating a data source collection, from a big class named MainViewModel and you move this functionality to a new class named ItemCreator, then this class ItemCreator is logically still part of the view model component.
On class level the functionality is now outside the MainViewModel class (while MainViewModel now has a strong reference to the new class, in order to invoke the code). On application level (architecture level), the functionality is still in the same component.

You can project this example onto the often proposed dialog service: extracting the dialog logic from the view model to a dedicated class named DialogService doesn't move the logic outside the view model component: the view model still depends on this extracted functionality.
The view model still participates in the UI logic e.g by explicitly invoking the "service" to control when dialog is shown and to control the dialog type itself (e.g., file open, folder select, color picker etc.).
This all requires knowledge of the UI's business details. Knowledge, that per definition does not belong into the view model component. Of course, such knowlegde introduces a coupling/dependency from the view model component to the view component.

Responsibilities simply don't change because you name a class DialogService instead of e.g. DialogViewModel.

The DialogService is therefore an anti-pattern, which hides the real problem: having implemented view model classes, that depend on UI and execute UI logic.

1.5 Does writing code-behind violates the MVVM design pattern?

MVVM is a design pattern and design patterns are per definition library independent, framework independent and language or compiler independent. Therefore, code-behind is not a topic when talking about MVVM.

The code-behind file is absolutely a valid context to write UI code. It's just another file that contains C# code. Code-behind means "a file with a .xaml.cs extension". It's also the only place for event handlers. And you don't want to stay away from events.

Why does the mantra "No code in code-behind" exist?
For people that are new to WPF, UWP or Xamarin, like those skilled and experienced developers coming from frameworks like WinForms, we have to stress that using XAML should be the preferred way to write UI code. Implementing a Style or DataTemplate using C# (e.g. in the code-behind file) is too complicated and produces code that is very difficult to read => difficult to understand => difficult to maintain.
XAML is just perfect for such tasks. The visually verbose markup style perfectly expresses the UI's structure. It does this far better, than for example C# could ever do. Despite markup languages like XAML may feel inferior to some or not worth learning it, it's definitely the first choice when implementing GUI. We should strive to write as much GUI code as possible using XAML.

But such considerations are absolutely irrelevant in terms of the MVVM design pattern.

Code-behind is simply a compiler concept, realized by the partial directive (in C#). That's why code-behind has nothing to do with any design pattern. That's why neither XAML nor C# can't have anything to do with any design pattern.

2 Solution

Like the OP correctly concludes:

"I don't really want to do this [open a file picker dialog] in my ViewModel(where 'Browse' is referenced via a DelegateCommand). Because I believe that goes against MVVM methodology.

2.1 Some fundamental considerations

• A dialog is a UI control: a view.
• The handling of a dialog control or a control in general e.g. showing/hiding is UI logic.
• MVVM requirement: the view model does not know about the existence of an UI or users. Because of this, a control flow that requires the view model to actively wait or call for user input, really requires some re-design: it is a critical violation and breaks the architectural boundaries dictated by MVVM.
• Showing a dialog requires knowledge about when to show it and when to close it.
• Showing the dialog requires to know about the UI and user, because the only reason to show a dialog is to interact with the user.
• Showing the dialog requires knowledge about the current UI context (in order to choose the appropriate dialog type).
• It is not the dependency on assemblies or classes like OpenFileDialog or UIElement that breaks the MVVM pattern, but the implementation or reference of UI logic in the view model component or model component (although such a dependency can be a valuable hint).
• For the same reasons, it would be wrong to show the dialog from the model component too.
• The only component responsible for UI logic is the view component.
• From an MVVM point of view, there is nothing like C#, XAML, C++ or VB.NET. Which means, there is nothing like partial or the related infamous code-behind file (*.xaml.cs). The concept of code-behind exists to allow the compiler to merge the XAML part of a class with its C# part. After that merge, both files are treated as a single class: it's a pure compiler concept. partial is the magic that enables to write class code using XAML (the true compiler language is still C# or any other IL compliant language).
• ICommand is an interface of the .NET library and therefore not a topic when talking about MVVM. It's wrong to believe that every action has to be triggered by an ICommand implementation in the view model.
  Events are still a very useful concept that conform with MVVM, as long as the unidirectional dependency between the components is maintained. Always forcing the use of ICommand instead of using events leads to unnatural and smelly code like the code presented by the OP.
• There is no such "rule" that ICommand must only be implemented by a view model class. It can be implemented by a view class too.
  In fact, views commonly implement RoutedCommand (or RoutedUICommand), which both are implementions of ICommand, and can also be used to trigger the display of a dialog from e.g., a Window or any other control.
  We have data binding to allow the UI to exchange data with the view model (anonymously, from the data source point of view). But since data binding can't invoke operations (at least in WPF - e.g., UWP allows this), we have ICommand and ICommandSource to realize this.
• Interfaces in general are not a relevant concept of MVVM. Therefore, introducing an interface (e.g., IFileDialogService) can never solve a MVVM related problem.
• Services or helper classes are not a concept of MVVM. Therefore, introducing services or helper classes can never solve a MVVM related problem.
• Classes an their names or type names in general are not relevant in terms of MVVM. Moving view model code to a separate class, even if that class is not named or suffixed with ViewModel, can't solve a MVVM related problem.

2.2 Conclusion

The solution is to show the OpenFileDialog from a class, that is part of the view component.
This means, such a class must be a class that is unknown to the view model and therefore can't be invoked by the view model.

This logic could be implemented directly in the code-behind file or inside any other class (file). The implementation can be a simple helper class or a more sophisticated (attached) behavior.

The point is: the dialog i.e. the UI component must be handled by the view component alone, as this is the only component that contains UI related logic. Since the view model does not have any knowledge of a view, it can't act actively to communicate with the view. Only passive communication is allowed (data binding, events).

We can always implement a certain flow using events raised by the view model that can be observed by the view in order to take actions like interacting with the user using a dialog.

There exist solutions using the view-model-first approach, which is does not violate MVVM in the first place. But still, badly designed responsibilities can turn this solution into an anti-pattern too.

3 How to fix the need for certain dialog requests

Most of the times, we can eliminate the need to show dialogs from within the application by fixing the application's design.

Since dialogs are a UI concept to enable interaction with the user, we must evaluate dialogs using UI design rules.
Maybe the most famous design rules for UI design are the 10 rules postulated by Nielsen and Molich in the 90's.

One important rule is about error prevention: it states that

a) we must prevent any kind of errors, especially input related, because
b) the user does not like his productivity to be interrupted by error messages and dialogs.

a) means: input data validation. Don't allow invalid data to enter the business logic.
b) means: avoid showing dialogs to the user, whenever possible. Never show a dialog from within the application and let the user trigger dialogs explicitly e.g., on mouse click (no unexpected interruption).

Following this simple rule certainly always eliminates the need to show a dialog triggered by the view model.

From the user's perspective, an application is a black box: it outputs data, accepts data and processes the input data. If we control the data input to guard against invalid data, we eliminate undefined or illegal states and ensure data integrity. This would mean that there is no need to ever show a dialog to the user from inside the application. Only those explicitly triggered by the user.

For example, a common scenario is that our model needs to persist data in a file. If the destination file already exists, we want to ask the user to confirm to overwrite this file.

Following the rule of error prevention, we always let the user pick files in the first place: whether it is a source file or a destination file, it's always the user who specifies this file by explicitly picking it via a file dialog. This means, the user must also explicitly trigger the file operation, for example by clicking on a "Save As" button.

This way, we can use a file picker or file save dialog to ensure only existing files are selected. As a bonus, we additionally eliminate the need to warn the user about overwriting existing files.

Following this approach, we have satisfied a) "[...]prevent any kind of errors, especially input related" and b) "[...]the user does not like to be interrupted by error messages and dialogs".

Update

Since people are questioning the fact that you don't need a view model to handle the dialog views, by coming up with extra "complicated" requirements like data validation to proof their point, I am forced to provide more complex examples to address these more complex scenarios (that were not initially requested by the OP).

4 Examples

4.1 Overview

The scenario is a simple input form to collect an user input like an album name and then use the OpenFileDialog to pick a destination file where the album name is saved to.
Three simple solutions:

Solution 1: Very simple and basic scenario, that meets the exact requirements of the question. Solution 2: Solution that enables to use data validation in the view model. For brevity the implementation of INotifyDataErrorInfo is omitted.
Solution 3: Another, more elegant solution, that uses an ICommand and the ICommandSource.CommandParameter to send the dialog result to the view model and execute the operation.

Solution 1

The following example provides a simple and intuitive solution to show the OpenFileDialog in a MVVM compliant way.
The solution allows the view model to remain unaware of any UI components or logic.

You can even consider to pass a FileStream to the view model instead of the file path. This way, you can handle any errors, while creating the stream, directly in the UI e.g., by showing a dialog if needed.

View

MainWindow.xaml

<Window>
  <Window.DataContext>
    <MainViewModel />
  </Window.DataContext>

  <StackPanel>
    <!-- The data to persist -->
    <TextBox Text="{Binding AlbumName}" />

    <!-- Show the file dialog. 
         Let user explicitly trigger the file save operation. 
         This button will be disabled until the required input is valid -->
    <Button Content="Save as" 
            Click="AppendAlbumNameToFile_OnClick" />
  </StackPanel>
</Window>

MainWindow.xaml.cs

partial class MainWindow : Window
{
  public MainWindow()
  {
    InitializeComponent();
  }

  private void AppendAlbumNameToFile_OnClick(object sender, EventArgs e)
  {
    var dialog = new OpenFileDialog();

    if (dialog.ShowDialog() == true)
    {
      // Consider to create the FileStream here to handle errors 
      // related to the user's picked file in the view. 
      // If opening the FileStream succeeds, we can pass it over to the viewmodel.
      string destinationFilePath = dialog.FileName;
      (this.DataContext as MainViewModel)?.SaveAlbumName(destinationFilePath);
    }
  }
}

View Model

MainViewModel.cs

class MainViewModel : INotifyPropertyChanged
{
  private string albumName;
  public string AlbumName  
  {
    get => this.albumName;
    set
    {
      this.albumName = value;
      OnPropertyChanged();
    }
  }
    
  // A model class that is responsible to persist and load data
  private DataRepository DataRepository { get; }
    
  // Default constructor
  public MainViewModel() => this.DataRepository = new DataRepository();
    
  // Since 'destinationFilePath' was picked using a file dialog, 
  // this method can't fail. If it fails, the error is not related to the user input
  public void SaveAlbumName(string destinationFilePath)
  {
    // Use a aggregated/composed model class to persist the data
    this.DataRepository.SaveData(this.AlbumName, destinationFilePath);
  }
}

Solution 2

A more realistic solution is to add a dedicated TextBox as input field to collect the destination file path.
A button can open the optional file picker view to allow the user to alternatively browse the filesystem for a destination path.

The browser result is assigned to the TextBox, which is bound to the view model class. This way the file path can be validated e.g., by implementing INotifyDataErrorInfo:

View

MainWindow.xaml

<Window>
  <Window.DataContext>
    <MainViewModel />
  </Window.DataContext>

  <StackPanel>
    <!-- The data to persist -->
    <TextBox Text="{Binding AlbumName}" />

    <!-- Alternative file path input, validated using INotifyDataErrorInfo validation 
         e.g. using File.Exists to validate the file path -->
    <TextBox x:Name="FilePathTextBox" 
             Text="{Binding DestinationPath, ValidatesOnNotifyDataErrors=True}" />

    <!-- Option to search a file using the file picker dialog -->
    <Button Content="Browse" Click="PickFile_OnClick" />

    <!-- Let user explicitly trigger the file save operation. 
         This button will be disabled until the required input is valid -->
    <Button Content="Save as" 
            Command="{Binding SaveAlbumNameCommand}" />
  </StackPanel>
</Window>

MainWindow.xaml.cs

partial class MainWindow : Window
{
  public MainWindow()
  {
    InitializeComponent();
  }

  private void PickFile_OnClick(object sender, EventArgs e)
  {
    var dialog = new OpenFileDialog();

    if (dialog.ShowDialog() == true)
    {
      this.FilePathTextBox.Text = dialog.FileName;

      // Since setting the property explicitly bypasses the data binding, 
      // we must explicitly update it by calling BindingExpression.UpdateSource()
      this.FilePathTextBox
        .GetBindingExpression(TextBox.TextProperty)
        .UpdateSource();
    }
  }
}

View Model

MainViewModel.cs

class MainViewModel : INotifyPropertyChanged, INotifyDataErrorInfo
{
  private string albumName;
  public string AlbumName
  {
    get => this.albumName;
    set
    {
      this.albumName = value;
      OnPropertyChanged();
    }
  }

  private string destinationPath;
  public string DestinationPath
  {
    get => this.destinationPath;
    set
    {
      this.destinationPath = value;
      OnPropertyChanged();

      ValidateDestinationFilePath();
    }
  }

  public ICommand SaveAlbumNameCommand => new RelayCommand(
    commandParameter =>
    {
      ExecuteSaveAlbumName(this.TextValue);
    },
    commandParameter => true);

  // A model class that is responsible to persist and load data
  private DataRepository DataRepository { get; }

  // Default constructor
  public MainViewModel() => this.DataRepository = new DataRepository();

  private void ExecuteSaveAlbumName(string destinationFilePath)
  {
    // Use a aggregated/composed model class to persist the data
    this.DataRepository.SaveData(this.AlbumName, destinationFilePath);
  }
}

Solution 3

The following solution is a more elegant version of the second scenario. It uses the ICommandSource.CommandParameter property to send the dialog result to the view model (instead of the data binding used in the previous example):

View

MainWindow.xaml

<Window x:Name="Window">
  <Window.DataContext>
    <MainViewModel />
  </Window.DataContext>

  <StackPanel>

    <!-- The data to persist -->
    <TextBox Text="{Binding AlbumName}" />

    <!-- Alternative file path input, validated using binding validation 
         e.g. using File.Exists to validate the file path -->
    <TextBox x:Name="FilePathTextBox" 
             Text="{Binding ElementName=Window, Path=DestinationPath, ValidationRules={Binding ElementName=Window, Path=FilePathValidationRules}" />

    <!-- Option to search a file using the file picker dialog -->
    <Button Content="Browse" Click="PickFile_OnClick" />

    <!-- Let user explicitly trigger the file save operation. 
         This button will be disabled until the required input is valid -->
    <Button Content="Save as" 
            CommandParameter="{Binding ElementName=Window, Path=DestinationPath}" 
            Command="{Binding SaveAlbumNameCommand}" />
  </StackPanel>
</Window>

MainWindow.xaml.cs

partial class MainWindow : Window
{
  public static readonly DependencyProperty DestinationPathProperty = DependencyProperty.Register(
    "DestinationPath",
    typeof(string),
    typeof(MainWindow),
    new PropertyMetadata(default(string)));

  public string DestinationPath
  {
    get => (string)GetValue(MainWindow.DestinationPathProperty);
    set => SetValue(MainWindow.DestinationPathProperty, value);
  }

  public MainWindow()
  {
    InitializeComponent();
  }

  private void PickFile_OnClick(object sender, EventArgs e)
  {
    var dialog = new OpenFileDialog();

    if (dialog.ShowDialog() == true)
    {
      this.DestinationPath = dialog.FileName;
    }
  }
}

View Model

MainViewModel.cs

class MainViewModel : INotifyPropertyChanged, INotifyDataErrorInfo
{
  private string albumName;
  public string AlbumName
  {
    get => this.albumName;
    set
    {
      this.albumName = value;
      OnPropertyChanged();
    }
  }

  public ICommand SaveAlbumNameCommand => new RelayCommand(
    commandParameter =>
    {
      ExecuteSaveAlbumName(commandParameter as string);
    },
    commandParameter => true);

  // A model class that is responsible to persist and load data
  private DataRepository DataRepository { get; }

  // Default constructor
  public MainViewModel() => this.DataRepository = new DataRepository();

  private void ExecuteSaveAlbumName(string destinationFilePath)
  {
    // Use a aggregated/composed model class to persist the data
    this.DataRepository.SaveData(this.AlbumName, destinationFilePath);
  }
}

I would have liked to comment on one of the answers, but alas, my reputation is not high enough to do so.

Having a call such as OpenFileDialog() violates the MVVM pattern because it implies a view (dialog) in the view model. The view model can call something like GetFileName() (that is, if simple binding is not sufficient), but it should not care how the file name is obtained.

The ViewModel should not open dialogs or even know of their existence. If the VM is housed in a separate DLL, the project should not have a reference to PresentationFramework.

I like to use a helper class in the view for common dialogs.

The helper class exposes a command (not an event) which the window binds to in XAML. This implies the use of RelayCommand within the view. The helper class is a DepencyObject so it can bind to the view model.

class DialogHelper : DependencyObject
{
    public ViewModel ViewModel
    {
        get { return (ViewModel)GetValue(ViewModelProperty); }
        set { SetValue(ViewModelProperty, value); }
    }

    public static readonly DependencyProperty ViewModelProperty =
        DependencyProperty.Register("ViewModel", typeof(ViewModel), typeof(DialogHelper),
        new UIPropertyMetadata(new PropertyChangedCallback(ViewModelProperty_Changed)));

    private static void ViewModelProperty_Changed(DependencyObject d, DependencyPropertyChangedEventArgs e)
    {
        if (ViewModelProperty != null)
        {
            Binding myBinding = new Binding("FileName");
            myBinding.Source = e.NewValue;
            myBinding.Mode = BindingMode.OneWayToSource;
            BindingOperations.SetBinding(d, FileNameProperty, myBinding);
        }
    }

    private string FileName
    {
        get { return (string)GetValue(FileNameProperty); }
        set { SetValue(FileNameProperty, value); }
    }

    private static readonly DependencyProperty FileNameProperty =
        DependencyProperty.Register("FileName", typeof(string), typeof(DialogHelper),
        new UIPropertyMetadata(new PropertyChangedCallback(FileNameProperty_Changed)));

    private static void FileNameProperty_Changed(DependencyObject d, DependencyPropertyChangedEventArgs e)
    {
        Debug.WriteLine("DialogHelper.FileName = {0}", e.NewValue);
    }

    public ICommand OpenFile { get; private set; }

    public DialogHelper()
    {
        OpenFile = new RelayCommand(OpenFileAction);
    }

    private void OpenFileAction(object obj)
    {
        OpenFileDialog dlg = new OpenFileDialog();

        if (dlg.ShowDialog() == true)
        {
            FileName = dlg.FileName;
        }
    }
}

The helper class needs a reference to the ViewModel instance. See the resource dictionary. Just after construction, the ViewModel property is set (in the same line of XAML). This is when the FileName property on the helper class is bound to the FileName property on the view model.

<Window x:Class="DialogExperiment.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:local="clr-namespace:DialogExperiment"
        xmlns:vm="clr-namespace:DialogExperimentVM;assembly=DialogExperimentVM"
        Title="MainWindow" Height="350" Width="525">
    <Window.Resources>
        <vm:ViewModel x:Key="viewModel" />
        <local:DialogHelper x:Key="helper" ViewModel="{StaticResource viewModel}"/>
    </Window.Resources>
    <DockPanel DataContext="{StaticResource viewModel}">
        <Menu DockPanel.Dock="Top">
            <MenuItem Header="File">
                <MenuItem Header="Open" Command="{Binding Source={StaticResource helper}, Path=OpenFile}" />
            </MenuItem>
        </Menu>
    </DockPanel>
</Window>

I use a service which i for example can pass into the constructor of my viewModel or resolve via dependency injection. e.g.

public interface IOpenFileService
{
    string FileName { get; }
    bool OpenFileDialog()
}

and a class implementing it, using OpenFileDialog under the hood. In the viewModel, i only use the interface and thus can mock/replace it if needed.